Navigating in snow

ABSTRACT

Systems and methods navigate a vehicle on a road at least partially covered with snow. In one implementation, a system may include at least one processor programmed to receive from an image capture device, a plurality of images captured of an environment forward of the vehicle, including an area where snow covers a road on which the vehicle travels, analyze at least one of the plurality of images to identify a first free space boundary on a first side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger side of the vehicle and extending forward of the vehicle, and a forward free space boundary forward of the vehicle and extending between the first free space boundary and the second free space boundary.

CROSS REFERENCES TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 15/673,343,filed on Aug. 9, 2017, which is a continuation of PCT InternationalApplication No. PCT/US2016/017411, filed Feb. 10, 2016, which claims thebenefit of priority of U.S. Provisional Patent Application No.62/114,091, filed on Feb. 10, 2015; U.S. Provisional Patent ApplicationNo. 62/164,055, filed on May 20, 2015; U.S. Provisional PatentApplication No. 62/170,728, filed on Jun. 4, 2015; U.S. ProvisionalPatent Application No. 62/181,784, filed on Jun. 19, 2015; U.S.Provisional Patent Application No. 62/192,576, filed on Jul. 15, 2015;U.S. Provisional Patent Application No. 62/215,764, filed on Sep. 9,2015; U.S. Provisional Patent Application No. 62/219,733, filed on Sep.17, 2015; U.S. Provisional Patent Application No. 62/261,578, filed onDec. 1, 2015; U.S. Provisional Patent Application No. 62/261,598, filedon Dec. 1, 2015; U.S. Provisional Patent Application No. 62/267,643,filed on Dec. 15, 2015; U.S. Provisional Patent Application No.62/269,818, filed on Dec. 18, 2015; U.S. Provisional Patent ApplicationNo. 62/270,408, filed on Dec. 21, 2015; U.S. Provisional PatentApplication No. 62/270,418, filed on Dec. 21, 2015; U.S. ProvisionalPatent Application No. 62/270,431, filed on Dec. 21, 2015; U.S.Provisional Patent Application No. 62/271,103, filed on Dec. 22, 2015;U.S. Provisional Patent Application No. 62/274,883, filed on Jan. 5,2016; U.S. Provisional Patent Application No. 62/274,968, filed on Jan.5, 2016; U.S. Provisional Patent Application No. 62/275,007, filed onJan. 5, 2016; U.S. Provisional Patent Application No. 62/275,046, filedon Jan. 5, 2016; and U.S. Provisional Patent Application No. 62/277,068,filed on Jan. 11, 2016. All of the foregoing applications areincorporated herein by reference in their entirety.

BACKGROUND Technical Field

The present disclosure relates generally to autonomous vehiclenavigation and a sparse map for autonomous vehicle navigation.Additionally, this disclosure relates to systems and methods forconstructing, using, and updating the sparse map for autonomous vehiclenavigation.

Background Information

As technology continues to advance, the goal of a fully autonomousvehicle that is capable of navigating on roadways is on the horizon.Autonomous vehicles may need to take into account a variety of factorsand make appropriate decisions based on those factors to safely andaccurately reach an intended destination. For example, an autonomousvehicle may need to process and interpret visual information (e.g.,information captured from a camera) and may also use informationobtained from other sources (e.g., from a GPS device, a speed sensor, anaccelerometer, a suspension sensor, etc.). At the same time, in order tonavigate to a destination, an autonomous vehicle may also need toidentify its location within a particular roadway (e.g., a specific lanewithin a multi-lane road), navigate alongside other vehicles, avoidobstacles and pedestrians, observe traffic signals and signs, and travelfrom on road to another road at appropriate intersections orinterchanges. Harnessing and interpreting vast volumes of informationcollected by an autonomous vehicle as it travels to its destinationposes a multitude of design challenges. The sheer quantity of data(e.g., captured image data, map data, GPS data, sensor data, etc.) thatan autonomous vehicle may need to analyze, access, and/or store poseschallenges that can in fact limit or even adversely affect autonomousnavigation. Furthermore, if an autonomous vehicle relies on traditionalmapping technology to navigate, the sheer volume of data needed to storeand update the map poses daunting challenges.

SUMMARY

Embodiments consistent with the present disclosure provide systems andmethods for autonomous vehicle navigation. The disclosed embodiments mayuse cameras to provide autonomous vehicle navigation features. Forexample, consistent with the disclosed embodiments, the disclosedsystems may include one, two, or more cameras that monitor theenvironment of a vehicle. The disclosed systems may provide anavigational response based on, for example, an analysis of imagescaptured by one or more of the cameras. The navigational response mayalso take into account other data including, for example, globalpositioning system (GPS) data, sensor data (e.g., from an accelerometer,a speed sensor, a suspension sensor, etc.), and/or other map data.

In some embodiments, the disclosed systems and methods may use a sparsemap for autonomous vehicle navigation. For example, the sparse map mayprovide sufficient information for navigation without requiringexcessive data storage.

In other embodiments, the disclosed systems and methods may construct aroad model for autonomous vehicle navigation. For example, the disclosedsystems and methods may use crowd sourced data for autonomous vehiclenavigation including recommended trajectories. As other examples, thedisclosed systems and methods may identify landmarks in an environmentof a vehicle and refine landmark positions.

In yet other embodiments, the disclosed systems and methods may use asparse road model for autonomous vehicle navigation. For example, thedisclosed systems and methods may provide navigation based on recognizedlandmarks, align a vehicle's tail for navigation, allow a vehicle tonavigate road junctions, allow a vehicle to navigate using localoverlapping maps, allow a vehicle to navigate using a sparse map,navigate based on an expected landmark location, autonomously navigate aroad based on road signatures, provide forward navigation based on arearward facing camera, navigate based on a free space determination,navigate in snow, provide autonomous vehicle speed calibration,determine lane assignment based on a recognized landmark location, anduse super landmarks as navigation aids.

In still yet other embodiments, the disclosed systems and methods mayprovide adaptive autonomous navigation. For example, disclosed systemsand methods may provide adaptive navigation based on user intervention,provide self-aware adaptive navigation, provide an adaptive road modelmanager, and manage a road model based on selective feedback.

In some embodiments, a non-transitory computer-readable medium mayinclude a sparse map for autonomous vehicle navigation along a roadsegment. The sparse map may include a polynomial representation of atarget trajectory for the autonomous vehicle along the road segment; anda plurality of predetermined landmarks associated with the road segment,wherein the plurality of predetermined landmarks may be spaced apart byat least 50 meters, and wherein the sparse map may have a data densityof no more than 1 megabyte per kilometer.

In some embodiments of the non-transitory computer-readable medium, thepolynomial representation may be a three-dimensional polynomialrepresentation. The polynomial representation of the target trajectorymay be determined based on two or more reconstructed trajectories ofprior traversals of vehicles along the road segment. The plurality ofpredetermined landmarks may include a traffic sign represented in thesparse map by no more than 50 bytes of data. The plurality ofpredetermined landmarks may include a directional sign represented inthe sparse map by no more than 50 bytes of data. The plurality ofpredetermined landmarks may include a general purpose sign representedin the sparse map by no more than 100 bytes of data. The plurality ofpredetermined landmarks may include a generally rectangular objectrepresented in the sparse map by no more than 100 bytes of data. Therepresentation of the generally rectangular object in the sparse map mayinclude a condensed image signature associated with the generallyrectangular object. The plurality of predetermined landmarks may berepresented in the sparse map by parameters including landmark size,distance to previous landmark, landmark type, and landmark position. Theplurality of predetermined landmarks included in the sparse map may bespaced apart by at least 2 kilometers. The plurality of predeterminedlandmarks included in the sparse map may be spaced apart by at least 1kilometer. The plurality of predetermined landmarks included in thesparse map may be spaced apart by at least 100 meters. The sparse mapmay have a data density of no more than 100 kilobytes per kilometer. Thesparse map may have a data density of no more than 10 kilobytes perkilometer. The plurality of predetermined landmarks may appear in thesparse map at a rate that is above a rate sufficient to maintain alongitudinal position determination accuracy within 1 meter.

In some embodiments, an autonomous vehicle may include a body; and anon-transitory computer-readable medium that may include a sparse mapfor autonomous vehicle navigation along a road segment. The sparse mapmay include a polynomial representation of a target trajectory for theautonomous vehicle along the road segment; and a plurality ofpredetermined landmarks associated with the road segment, wherein theplurality of predetermined landmarks are spaced apart by at least 50meters, and wherein the sparse map has a data density of no more than 1megabyte per kilometer. The autonomous vehicle may include a processorconfigured to execute data included in the sparse map for providingautonomous vehicle navigation along the road segment.

In some embodiments of the autonomous vehicle, the polynomialrepresentation may be a three-dimensional polynomial representation. Thepolynomial representation of the target trajectory may be determinedbased on two or more reconstructed trajectories of prior traversals ofvehicles along the road segment.

In some embodiments, an autonomous vehicle may include a body; and aprocessor configured to receive data included in a sparse map andexecute the data for autonomous vehicle navigation along a road segment.The sparse map may include a polynomial representation of a targettrajectory for the autonomous vehicle along the road segment; and aplurality of predetermined landmarks associated with the road segment,wherein the plurality of predetermined landmarks are spaced apart by atleast 50 meters, and wherein the sparse map has a data density of nomore than 1 megabyte per kilometer.

In some embodiments, a method of processing vehicle navigationinformation for use in autonomous vehicle navigation may includereceiving, by a server, navigation information from a plurality ofvehicles. The navigation information from the plurality of vehicles maybe associated with a common road segment. The method may includestoring, by the server, the navigation information associated with thecommon road segment. The method may include generating, by the server,at least a portion of an autonomous vehicle road navigation model forthe common road segment based on the navigation information from theplurality of vehicles; and distributing, by the server, the autonomousvehicle road navigation model to one or more autonomous vehicles for usein autonomously navigating the one or more autonomous vehicles along thecommon road segment.

In some embodiments of the method, the navigation information mayinclude a trajectory from each of the plurality of vehicles as eachvehicle travels over the common road segment. The trajectory may bedetermined based on sensed motion of a camera, includingthree-dimensional translation and three-dimensional rotational motions.The navigation information may include a lane assignment. Generating atleast a portion of the autonomous vehicle road navigation model mayinclude clustering vehicle trajectories along the common road segmentand determining a target trajectory along the common road segment basedon the clustered vehicle trajectories. The autonomous vehicle roadnavigation model may include a three-dimensional spline corresponding tothe target trajectory along the common road segment. The targettrajectory may be associated with a single lane of the common roadsegment. The autonomous vehicle road navigation model may include aplurality of target trajectories, each associated with a separate laneof the common road segment. Determining the target trajectory along thecommon road segment based on the clustered vehicle trajectories mayinclude finding a mean or average trajectory based on the clusteredvehicle trajectories. The target trajectory may be represented by athree-dimensional spline. The spline may be defined by less than 10kilobytes per kilometer. The autonomous vehicle road navigation modelmay include identification of at least one landmark, including aposition of the at least one landmark. The position of the at least onelandmark may be determined based on position measurements performedusing sensor systems associated with the plurality of vehicles. Theposition measurements may be averaged to obtain the position of the atleast one landmark. The at least one landmark may include at least oneof a traffic sign, an arrow marking, a lane marking, a dashed lanemarking, a traffic light, a stop line, a directional sign, a landmarkbeacon, or a lamppost.

In some embodiments, a navigation system for a vehicle may include atleast one processor programmed to receive from a camera, at least oneenvironmental image associated with the vehicle; analyze the at leastone environmental image to determine navigation information related tothe vehicle; transmit the navigation information from the vehicle to aserver. The at least one processor may be programmed to receive, fromthe server, an autonomous vehicle road navigation model. The autonomousvehicle road navigation model may include at least one update based onthe transmitted navigation information. The at least one processor maybe programmed to cause at least one navigational maneuver by the vehiclebased on the autonomous vehicle road navigation model.

In some embodiments of the navigation system, the navigation informationmay include a trajectory from each of the plurality of vehicles as eachvehicle travels over the common road segment.

In some embodiments, a server for processing vehicle navigationinformation for use in autonomous vehicle navigation may include acommunication unit configured to communicate with a plurality ofvehicles; and at least one processor programmed to receive, via thecommunication unit, the navigation information from the vehicles. The atleast one processor may be programmed to generate at least a portion ofan autonomous vehicle road navigation model based on the navigationinformation; and transmit at least the portion of the autonomous vehicleroad navigation model to at least one of the vehicles to cause anavigational maneuver by the at least one of the vehicles based on theportion of the autonomous vehicle road navigation model.

In some embodiments of the server, the navigation information mayinclude a trajectory from each of the plurality of vehicles as eachvehicle travels over the common road segment. The portion of autonomousvehicle road navigation model may include an update to the autonomousvehicle road navigation model.

In some embodiments, a navigation system for a vehicle may include atleast one processor programmed to receive, from one or more sensors,outputs indicative of a motion of the vehicle; determine an actualtrajectory of the vehicle based on the outputs from the one or moresensors; receive, from a camera, at least one environmental imageassociated with the vehicle; analyze the at least one environmentalimage to determine information associated with at least one navigationalconstraint; determine a target trajectory, including the actualtrajectory of the vehicle and one or more modifications to the actualtrajectory based on the determined information associated with the atleast one navigational constraint; and transmit the target trajectoryfrom the vehicle to a server.

In some embodiments of the system, the one or more sensors may include aspeed sensor. The one or more sensors may include an accelerometer. Theone or more sensors may include the camera. The at least onenavigational constraint may include at least one of a barrier, anobject, a lane marking, a sign, or another vehicle. The camera may beincluded in the vehicle.

In some embodiments, a method of uploading a target trajectory to aserver may include receiving, from one or more sensors, outputsindicative of a motion of a vehicle; determining an actual trajectory ofthe vehicle based on the outputs from the one or more sensors;receiving, from a camera, at least one environmental image associatedwith the vehicle; analyzing the at least one environmental image todetermine information associated with at least one navigationalconstraint; determining a target trajectory, including the actualtrajectory of the vehicle and one or more modifications to the actualtrajectory based on the determined information associated with the atleast one navigational constraint; and transmitting the targettrajectory from the vehicle to a server.

In some embodiments of the method, the one or more sensors may include aspeed sensor. The one or more sensors may include an accelerometer. Theone or more sensors may include the camera. The at least onenavigational constraint may include at least one of a barrier, anobject, a lane marking, a sign, or another vehicle. The camera may beincluded in the vehicle.

In some embodiments, a system for identifying a landmark for use inautonomous vehicle navigation may include at least one processorprogrammed to: receive at least one identifier associated with thelandmark; associate the landmark with a corresponding road segment;update an autonomous vehicle road navigation model relative to thecorresponding road segment to include the at least one identifierassociated with the landmark; and distribute the updated autonomousvehicle road navigation model to a plurality of autonomous vehicles. Theat least one identifier may be determined based on acquisition, from acamera associated with a host vehicle, of at least one imagerepresentative of an environment of the host vehicle; analysis of the atleast one image to identify the landmark in the environment of the hostvehicle; and analysis of the at least one image to determine the atleast one identifier associated with the landmark.

In some embodiments of the system, the at least one identifier mayinclude a position of the landmark. The at least one identifier mayinclude a shape of the landmark. The at least one identifier may includea size of the landmark. The at least one identifier may include adistance of the landmark relative to another landmark. The at least oneidentifier may be determined based on the landmark being identified asone of a plurality of landmark types. The landmark types may include atraffic sign. The landmark types may include a post. The landmark typesmay include a directional indicator. The landmark types may include arectangular sign. The at least one identifier further may include acondensed signature representation. The condensed signaturerepresentation of the landmark may be determined based on mapping animage of the landmark to a sequence of numbers of a predetermined datasize. The condensed signature representation may indicate an appearanceof the landmark. The condensed signature representation may indicate atleast one of a color pattern of an image of the landmark or a brightnesspattern of the image. The landmark may include at least one of adirectional sign, a traffic sign, a lamppost, a road marking, and abusiness sign.

In some embodiments, a method of identifying a landmark for use inautonomous vehicle navigation may include receiving at least oneidentifier associated with the landmark; associating the landmark with acorresponding road segment; updating an autonomous vehicle roadnavigation model relative to the corresponding road segment to includethe at least one identifier associated with the landmark; anddistributing the updated autonomous vehicle road navigation model to aplurality of autonomous vehicles.

In some embodiments, the method may include determining the at least oneidentifier. Determining the at least one identifier may includeacquiring, from a camera associated with a host vehicle, at least oneimage representative of an environment of the host vehicle; analyzingthe at least one image to identify the landmark in the environment ofthe host vehicle; and analyzing the at least one image to determine theat least one identifier associated with the landmark. The at least oneidentifier may include a distance of the landmark relative to anotherlandmark, and wherein determining the at least one identifier includesdetermining a distance of the landmark relative to another landmark. Theat least one identifier may further include a condensed signaturerepresentation, and wherein determining the at least one identifierincludes determining the condensed signature representation from the atleast one image.

In some embodiments, a system for determining a location of a landmarkfor use in navigation of an autonomous vehicle may include at least oneprocessor programmed to: receive a measured position of the landmark;and determine a refined position of the landmark based on the measuredposition of the landmark and at least one previously acquired positionfor the landmark. The measured position and the at least one previouslyacquired position may be determined based on acquisition, from a cameraassociated with a host vehicle, of at least one environmental imageassociated with the host vehicle, analysis of the at least oneenvironmental image to identify the landmark in the environment of thehost vehicle, reception of global positioning system (GPS) datarepresenting a location of the host vehicle, analysis of the at leastone environmental image to determine a relative position of theidentified landmark with respect to the host vehicle, and determinationof a globally localized position of the landmark based on at least theGPS data and the determined relative position.

In some embodiments of the system, the landmark may include at least oneof a traffic sign, an arrow, a lane marking, a dashed lane marking, atraffic light, a stop line, a directional sign, a landmark beacon, or alamppost. Analysis of the at least one image to determine the relativeposition of the identified landmark with respect to the vehicle mayinclude calculating a distance based on a scale associated with the atleast one image. Analyzing the at least one image to determine therelative position of the identified landmark with respect to the vehiclemay include calculating a distance based on an optical flow associatedwith the at least one image. The GPS data may be received from a GPSdevice included in the host vehicle. The camera may be included in thehost vehicle. Determining the refined position of the landmark mayinclude averaging the measured position of the landmark with the atleast one previously acquired position.

In some embodiments, a method for determining a location of a landmarkfor use in navigation of an autonomous vehicle may include receiving ameasured position of the landmark; and determining a refined position ofthe landmark based on the measured position of the landmark and at leastone previously acquired position for the landmark. The measured positionand the at least one previously acquired position may be determinedbased on acquisition, from a camera associated with a host vehicle, ofat least one environmental image associated with the host vehicle,analysis of the at least one environmental image to identify thelandmark in the environment of the host vehicle, reception of globalpositioning system (GPS) data representing a location of the hostvehicle, analysis of the at least one environmental image to determine arelative position of the identified landmark with respect to the hostvehicle, and determination of a globally localized position of thelandmark based on at least the GPS data and the determined relativeposition.

In some embodiments of the method, the landmark may include at least oneof a traffic sign, an arrow, a lane marking, a dashed lane marking, atraffic light, a stop line, a directional sign, a landmark beacon, or alamppost. Analysis of the at least one image to determine the relativeposition of the identified landmark with respect to the vehicle mayinclude calculating a distance based on a scale associated with the atleast one image. Analysis of the at least one image to determine therelative position of the identified landmark with respect to the vehiclemay include calculating a distance based on an optical flow associatedwith the at least one image. The GPS data may be received from a GPSdevice included in the host vehicle. The camera may be included in thehost vehicle. Determining the refined position of the landmark mayinclude averaging the measured position of the landmark with the atleast one previously acquired position.

In some embodiments, an autonomous vehicle may include a body and atleast one processor programmed to receive a measured position of thelandmark; and determine a refined position of the landmark based on themeasured position of the landmark and at least one previously acquiredposition for the landmark. The at least one processor may be furtherprogrammed to determine the measured position and the at least onepreviously acquired position based on acquisition, from a cameraassociated with the vehicle, of at least one environmental imageassociated with the vehicle, analysis of the at least one environmentalimage to identify the landmark in the environment of the vehicle,reception of global positioning system (GPS) data representing alocation of the vehicle, analysis of the at least one environmentalimage to determine a relative position of the identified landmark withrespect to the vehicle, and determination of a globally localizedposition of the landmark based on at least the GPS data and thedetermined relative position.

In some embodiments of the vehicle, the landmark may include at leastone of a traffic sign, an arrow, a lane marking, a dashed lane marking,a traffic light, a stop line, a directional sign, a landmark beacon, ora lamppost. Analysis of the at least one image to determine the relativeposition of the identified landmark with respect to the vehicle mayinclude calculating a distance based on a scale associated with the atleast one image. Analyzing the at least one image to determine therelative position of the identified landmark with respect to the vehiclemay include calculating a distance based on an optical flow associatedwith the at least one image. The GPS data may be received from a GPSdevice included in the host vehicle. Determining the refined position ofthe landmark may include averaging the measured position of the landmarkwith the at least one previously acquired position

In some embodiments, a system for autonomously navigating a vehiclealong a road segment may include at least one processor programmed to:receive from an image capture device at least one image representativeof an environment of the vehicle; analyze the at least one image toidentify at least one recognized landmark; determine a current locationof the vehicle relative to a predetermined road model trajectoryassociated with the road segment based, at least in part, on apredetermined location of the recognized landmark; and determine anautonomous steering action for the vehicle based on a direction of thepredetermined road model trajectory at the determined current locationof the vehicle relative to the predetermined road model trajectory.

In some embodiments of the system, the recognized landmark may includeat least one of a traffic sign, an arrow marking, a lane marking, adashed lane marking, a traffic light, a stop line, a directional sign, areflector, a landmark beacon, or a lamppost. The recognized landmark mayinclude a change in spacing of lines on the road segment. The recognizedlandmark may include a sign for a business. The predetermined road modeltrajectory may include a three-dimensional polynomial representation ofa target trajectory along the road segment. Navigation betweenrecognized landmarks may include integration of vehicle velocity todetermine a location of the vehicle along the predetermined road modeltrajectory. The processor may be further programmed to adjust a steeringsystem of the vehicle based on the autonomous steering action tonavigate the vehicle. The processor may be further programmed to:determine a distance of the vehicle from the at least one recognizedlandmark; and determine whether the vehicle is positioned on thepredetermined road model trajectory associated with the road segmentbased on the distance. The processor may be further programmed to adjustthe steering system of the vehicle to move the vehicle from a currentposition of the vehicle to a position on the predetermined road modeltrajectory when the vehicle is not positioned on the predetermined roadmodel trajectory.

In some embodiments, a vehicle may include a body; at least one imagecapture device configured to acquire at least one image representativeof an environment of the vehicle; and at least one processor programmedto: receive from the at least one image capture device the at least oneimage; analyze the at least one image to identify at least onerecognized landmark; determine a current location of the vehiclerelative to a predetermined road model trajectory associated with theroad segment based, at least in part, on a predetermined location of therecognized landmark; and determine an autonomous steering action for thevehicle based on a direction of the predetermined road model trajectoryat the determined current location of the vehicle relative to thepredetermined road model trajectory.

In some embodiments of the vehicle, the recognized landmark may includeat least one of a traffic sign, an arrow marking, a lane marking, adashed lane marking, a traffic light, a stop line, a directional sign, areflector, a landmark beacon, a lamppost, a change is spacing of lineson the road, or a sign for a business. The predetermined road modeltrajectory may include a three-dimensional polynomial representation ofa target trajectory along the road segment. Navigation betweenrecognized landmarks may include integration of vehicle velocity todetermine a location of the vehicle along the predetermined road modeltrajectory. The processor may be further programmed to adjust thesteering system of the vehicle based on the autonomous steering actionto navigate the vehicle. The processor may be further programmed to:determine a distance of the vehicle from the at least one recognizedlandmark; and determine whether the vehicle is positioned on thepredetermined road model trajectory associated with the road segmentbased on the distance. The processor may be further programmed to adjustthe steering system of the vehicle to move the vehicle from a currentposition of the vehicle to a position on the predetermined road modeltrajectory when the vehicle is not positioned on the predetermined roadmodel trajectory.

In some embodiments, a method of navigating a vehicle may includereceiving, from an image capture device associated with the vehicle, atleast one image representative of an environment of the vehicle;analyzing, using a processor associated with the vehicle, the at leastone image to identify at least one recognized landmark; determining acurrent position of the vehicle relative to a predetermined road modeltrajectory associated with the road segment based, at least in part, ona predetermined location of the recognized landmark; determining anautonomous steering action for the vehicle based on a direction of thepredetermined road model trajectory at the determined current locationof the vehicle relative to the predetermined road model trajectory; andadjusting a steering system of the vehicle based on the autonomoussteering action to navigate the vehicle.

In some embodiments, the method may include determining a location ofthe vehicle along the predetermined road model trajectory by integratingthe vehicle velocity. The method may include determining, using theprocessor, a distance of the vehicle from the at least one recognizedlandmark; and determining whether the vehicle is positioned on thepredetermined road model trajectory associated with the road segmentbased on the distance. The method may include determining atransformation required to move the vehicle from a current position ofthe vehicle to a position on the predetermined road model trajectory;and adjusting the steering system of the vehicle based on thetransformation.

In some embodiments, a system for autonomously navigating an autonomousvehicle along a road segment may include at least one processorprogrammed to receive from an image capture device, a plurality ofimages representative of an environment of the autonomous vehicle;determine a traveled trajectory of the autonomous vehicle along the roadsegment based, at least in part, on analysis of one or more of theplurality of images; determine a current location of the autonomousvehicle along a predetermined road model trajectory based on analysis ofone or more of the plurality of images; determine a heading directionfor the autonomous vehicle based on the determined traveled trajectory;and determine a steering direction for the autonomous vehicle, relativeto the heading direction, by comparing the traveled trajectory to thepredetermined road model trajectory at the current location of theautonomous vehicle.

In some embodiments of the system, the comparison between the traveledtrajectory and the predetermined road model trajectory may includedetermination of a transformation that reduces an error between thetraveled trajectory and the predetermined road model trajectory. Theprocessor may be further programmed to adjust the steering system of theautonomous vehicle based on the transformation. The predetermined roadmodel trajectory may include a three-dimensional polynomialrepresentation of a target trajectory along the road segment. Thepredetermined road model trajectory may be retrieved from a databasestored in a memory included in the autonomous vehicle. The predeterminedroad model trajectory may be retrieved from a database accessible to theautonomous vehicle over a wireless communications interface. The imagecapture device may be included in the autonomous vehicle. Determinationof the steering direction may be further based on one or more additionalcues, including one or more of a left lane mark polynomial model, aright lane mark polynomial model, holistic path prediction, motion of aforward vehicle, determined free space ahead of the autonomous vehicle,and virtual lanes or virtual lane constraints determined based onpositions of vehicles forward of the autonomous vehicle. Determinationof the steering direction may be based on weights applied to the one ormore additional cues.

In some embodiments, an autonomous vehicle may include a body; at leastone image capture device configured to acquire at least one imagerepresentative of an environment of the autonomous vehicle; and at leastone processor programmed to: receive from the image capture device, aplurality of images representative of the environment of the autonomousvehicle; determine a traveled trajectory of the autonomous vehicle alongthe road segment based, at least in part, on analysis of one or more ofthe plurality of images; determine a current location of the autonomousvehicle along a predetermined road model trajectory based on analysis ofone or more of the plurality of images; determine a heading directionfor the autonomous vehicle based on the determined traveled trajectory;and determine a steering direction for the autonomous vehicle, relativeto the heading direction, by comparing the traveled trajectory to thepredetermined road model trajectory at the current location of theautonomous vehicle.

In some embodiments of the autonomous vehicle, the comparison betweenthe traveled trajectory and the predetermined road model trajectory mayinclude determination of a transformation that reduces an error betweenthe traveled trajectory and the predetermined road model trajectory. Thepredetermined road model trajectory may include a three-dimensionalpolynomial representation of a target trajectory along the road segment.The predetermined road model trajectory may be retrieved from one of adatabase stored in a memory included in the autonomous vehicle and adatabase accessible to the autonomous vehicle over a wirelesscommunications interface. Determination of the steering direction may befurther based on one or more additional cues, including one or more of aleft lane mark polynomial model, a right lane mark polynomial model,holistic path prediction, motion of a forward vehicle, determined freespace ahead of the autonomous vehicle, and virtual lanes or virtual laneconstraints determined based on positions of vehicles forward of theautonomous vehicle. Determination of the steering direction may be basedon weights applied to the one or more additional cues

In some embodiments, a method of navigating an autonomous vehicle mayinclude receiving, from an image capture device, a plurality of imagesrepresentative of an environment of the autonomous vehicle; determininga traveled trajectory of the autonomous vehicle along the road segmentbased, at least in part, on analysis of one or more of the plurality ofimages; determining a current location of the autonomous vehicle along apredetermined road model trajectory based on analysis of one or more ofthe plurality of images; determining a heading direction for theautonomous vehicle based on the determined traveled trajectory; anddetermining a steering direction for the autonomous vehicle, relative tothe heading direction, by comparing the traveled trajectory to thepredetermined road model trajectory at the current location of theautonomous vehicle.

In some embodiments of the method, comparing the traveled trajectory tothe predetermined road model trajectory may include determining atransformation that reduces an error between the traveled trajectory andthe predetermined road model trajectory. Determining a steeringdirection may be based on one or more additional cues, including one ormore of a left lane mark polynomial model, a right lane mark polynomialmodel, holistic path prediction, motion of a forward vehicle, determinedfree space ahead of the autonomous vehicle, and virtual lanes or virtuallane constraints determined based on positions of vehicles forward ofthe autonomous vehicle. Determining the steering direction may includeapplying weights to the one or more additional cues.

In some embodiments, a system for autonomously navigating a vehiclethrough a road junction may include at least one processor programmedto: receive from an image capture device at least one imagerepresentative of an environment of the vehicle; analyze the at leastone image to identify two or more landmarks located in the environmentof the vehicle; determine, for each of the two or more landmarks, adirectional indicator relative to the vehicle; determine a currentlocation of the vehicle relative to the road junction based on anintersection of the directional indicators for the two or morelandmarks; determine a heading for the vehicle based on the directionalindicators for the two or more landmarks; and determine a steering anglefor the vehicle by comparing the vehicle heading with a predeterminedroad model trajectory at the current location of the vehicle.

In some embodiments of the system, the predetermined road modeltrajectory may include a three-dimensional polynomial representation ofa target trajectory along the road segment. The two or more landmarksmay include three or more landmarks. The at least one processor may befurther programmed to transmit a control signal specifying the steeringangle to a steering system of the vehicle. The processor may beconfigured to retrieve the predetermined road model trajectory from adatabase stored in a memory included in the vehicle. The processor maybe configured to retrieve the predetermined road model trajectory from adatabase accessible to the vehicle over a wireless communicationsinterface. The camera may be included in the vehicle. The processor maybe further programmed to determine the heading for the vehicle by:determining a previous location of the vehicle relative to the roadjunction based on the intersection of the directional indicators for thetwo or more landmarks; and determining the heading based on the previouslocation and the current location.

In some embodiments, an autonomous vehicle may include a body; at leastone image capture device configured to acquire at least one imagerepresentative of an environment of the vehicle; and at least oneprocessor programmed to: receive from a camera at least one imagerepresentative of an environment of the vehicle; analyze the at leastone image to identify two or more landmarks located in the environmentof the vehicle; determine, for each of the two or more landmarks, adirectional indicator relative to the vehicle; determine a currentlocation of the vehicle relative to the road junction based on anintersection of the directional indicators for the two or morelandmarks; determine a heading for the vehicle based on the directionalindicators for the two or more landmarks; and determine a steering anglefor the vehicle by comparing the vehicle heading with a predeterminedroad model trajectory at the current location of the vehicle.

In some embodiments of the vehicle, the predetermined road modeltrajectory may include a three-dimensional polynomial representation ofa target trajectory along the road segment. The two or more landmarksmay include three or more landmarks. The at least one processor may befurther programmed to transmit a control signal specifying the steeringangle to a steering system of the vehicle. The predetermined road modeltrajectory may be retrieved from one of a database stored in a memoryincluded in the vehicle and a database accessible to the vehicle over awireless communications interface. The processor may be furtherprogrammed to determine a heading for the vehicle by: determining aprevious location of the vehicle relative to the road junction based onthe intersection of the directional indicators for the two or morelandmarks; and determining the heading based on the previous locationand the current location.

In some embodiments, a method of navigating an autonomous vehicle mayinclude receiving, from an image capture device, at least one imagerepresentative of an environment of the vehicle; analyzing, using atleast one processor, the at least one image to identify two or morelandmarks located in the environment of the vehicle; determining, foreach of the two or more landmarks, a directional indicator relative tothe vehicle; determining a current location of the vehicle relative tothe road junction based on an intersection of the directional indicatorsfor the two or more landmarks; determining a heading for the vehiclebased on the directional indicators for the two or more landmarks; anddetermining a steering angle for the vehicle by comparing the vehicleheading with a predetermined road model trajectory at the currentlocation of the vehicle.

In some embodiments of the method, the predetermined road modeltrajectory may include a three-dimensional polynomial representation ofa target trajectory along the road segment. The method may includeretrieving the predetermined road model trajectory from one of adatabase stored in a memory included in the vehicle and a databaseaccessible to the vehicle over a wireless communications interface. Themethod may include transmitting a control signal specifying the steeringangle to a steering system of the vehicle. Determining the heading forthe vehicle may include determining a previous location of the vehiclerelative to the road junction based on the intersection of thedirectional indicators for the two or more landmarks; and determiningthe heading based on the previous location and the current location.

In some embodiments, a system for autonomously navigating a vehiclebased on a plurality of overlapping navigational maps may include atleast one processor programmed to: receive a first navigational map foruse in autonomously controlling the vehicle, wherein the firstnavigational map is associated with a first road segment; determine atleast a first autonomous navigational response for the vehicle along thefirst road segment based on analysis of the first navigational map;receive a second navigational map for use in autonomously controllingthe vehicle, wherein the second navigational map is associated with asecond road segment, wherein the first road segment is different fromthe second road segment, and wherein the first road segment and thesecond road segment overlap one another at an overlap segment; determineat least a second autonomous navigational response for the vehicle alongthe second road segment based on analysis of the second navigationalmap; and determine at least a third autonomous navigational response forthe vehicle in the overlap segment based on at least one of the firstnavigational map and the second navigational map.

In some embodiments of the system, each of the plurality of overlappingnavigational maps may have its own coordinate frame. Each of theplurality of overlapping navigational maps may include a polynomialrepresentation of a target trajectory along a road segment. Each of theoverlapping navigational maps may be a sparse map having a data densityof no more than 10 kilobytes per kilometer. The overlap segment may havea length of at least 50 meters. The overlap segment may have a length ofat least 100 meters. The at least one processor may be programmed todetermine the third autonomous navigational response based on both thefirst navigational map and the second navigational map. The thirdautonomous navigational response may be a combination of the firstautonomous navigational response and the second autonomous navigationalresponse. The third autonomous navigational response may be an averageof the first autonomous navigational response and the second autonomousnavigational response. The processor may be further programmed to:determine an error between the first autonomous navigational responseand the second autonomous navigational response; and determine the thirdautonomous navigational response based on the second autonomousnavigational response when the error is less than a threshold error.

In some embodiments, an autonomous vehicle may include a body; at leastone image capture device configured to acquire at least one imagerepresentative of an environment of the vehicle; at least one processorprogrammed to: determine a current location of the vehicle based on theat least one image; receive a first navigational map associated with afirst road segment; determine at least a first autonomous navigationalresponse for the vehicle based on analysis of the first navigationalmap, when the current location of the vehicle lies on the firstnavigational map; receive a second navigational map associated with asecond road segment different from the second road segment, the firstroad segment and the second road segment overlapping one another at anoverlap segment; determine at least a second autonomous navigationalresponse for the vehicle based on analysis of the second navigationalmap when the current location of the vehicle lies on the secondnavigational map; and determine at least a third autonomous navigationalresponse for the vehicle based on at least one of the first navigationalmap and the second navigational map when the current location of thevehicle lies in the overlap segment.

In some embodiments of the autonomous vehicle, each of the firstnavigational map and the second navigational map may have its owncoordinate frame. Each of the first navigational map and the secondnavigational map may include a polynomial representation of a targettrajectory along a road segment. The at least one processor may beprogrammed to determine the third autonomous navigational response basedon both the first navigational map and the second navigational map. Thethird autonomous navigational response may be a combination of the firstautonomous navigational response and the second autonomous navigationalresponse. The processor may be further programmed to: determine an errorbetween the first autonomous navigational response and the secondautonomous navigational response; and determine the third autonomousnavigational response based on the second autonomous navigationalresponse when the error is less than a threshold error.

In some embodiments, a method of navigating an autonomous vehicle mayinclude receiving from an image capture device, at least one imagerepresentative of an environment of the vehicle; determining, using aprocessor associated with the vehicle, a current location of the vehiclebased on the at least one image; receiving a first navigational mapassociated with a first road segment; determining at least a firstautonomous navigational response for the vehicle based on analysis ofthe first navigational map, when the current location of the vehiclelies on the first navigational map; receiving a second navigational mapassociated with a second road segment different from the second roadsegment, the first road segment and the second road segment overlappingone another at an overlap segment; determining at least a secondautonomous navigational response for the vehicle based on analysis ofthe second navigational map when the current location of the vehiclelies on the second navigational map; and determining at least a thirdautonomous navigational response for the vehicle based on at least oneof the first navigational map and the second navigational map when thecurrent location of the vehicle lies in the overlap segment.

In some embodiments of the method, each of the plurality of overlappingnavigational maps may have its own coordinate frame, and each of theplurality of overlapping navigational maps may include a polynomialrepresentation of a target trajectory along a road segment. Determiningthe third autonomous navigational response may include determining acombination of the first autonomous navigational response and the secondautonomous navigational response. The method may include determining anerror between the first autonomous navigational response and the secondautonomous navigational response; and determining the third autonomousnavigational response based on the second autonomous navigationalresponse when the error is less than a threshold error.

In some embodiments, a system for sparse map autonomous navigation of avehicle along a road segment may include at least one processorprogrammed to: receive a sparse map of the road segment, wherein thesparse map has a data density of no more than 1 megabyte per kilometer;receive from a camera, at least one image representative of anenvironment of the vehicle; analyze the sparse map and the at least oneimage received from the camera; and determine an autonomous navigationalresponse for the vehicle based solely on the analysis of the sparse mapand the at least one image received from the camera.

In some embodiments of the system, the sparse map may include apolynomial representation of a target trajectory along the road segment.The sparse map may include one or more recognized landmarks. Therecognized landmarks may be spaced apart in the sparse map at a rate ofno more than 0.5 per kilometer. The recognized landmarks may be spacedapart in the sparse map at a rate of no more than 1 per kilometer. Therecognized landmarks may be spaced apart in the sparse map at a rate ofno more than 1 per 100 meters. The sparse map may have a data density ofno more than 100 kilobytes per kilometer. The sparse map may have a datadensity of no more than 10 kilobytes per kilometer.

In some embodiments, a method for sparse map autonomous navigation of avehicle along a road segment may include receiving a sparse map of theroad segment, wherein the sparse map has a data density of no more than1 megabyte per kilometer; receiving from a camera, at least one imagerepresentative of an environment of the vehicle; analyzing the sparsemap and the at least one image received from the camera; and determiningan autonomous navigational response for the vehicle based solely on theanalysis of the sparse map and the at least one image received from thecamera.

In some embodiments of the method, the sparse map may include apolynomial representation of a target trajectory along the road segment.The sparse map may include one or more recognized landmarks. Therecognized landmarks may be spaced apart in the sparse map at a rate ofno more than 0.5 per kilometer. The recognized landmarks may be spacedapart in the sparse map at a rate of no more than 1 per kilometer. Therecognized landmarks may be spaced apart in the sparse map at a rate ofno more than 1 per 100 meters. The sparse map may have a data density ofno more than 100 kilobytes per kilometer. The sparse map may have a datadensity of no more than 10 kilobytes per kilometer.

In some embodiments, a non-transitory computer readable medium may storeinstructions causing at least one processor to perform sparse mapautonomous navigation of a vehicle along a road segment, which mayinclude receiving a sparse map of the road segment. The instructions maycause the processor to perform the steps of: receiving a sparse map ofthe road segment, wherein the sparse map has a data density of no morethan 1 megabyte per kilometer; receiving from a camera, at least oneimage representative of an environment of the vehicle; analyzing thesparse map and the at least one image received from the camera; anddetermining an autonomous navigational response for the vehicle basedsolely on the analysis of the sparse map and the at least one imagereceived from the camera.

In some embodiments of the non-transitory computer readable medium, thesparse map may include a polynomial representation of a targettrajectory along the road segment. The sparse map may include one ormore recognized landmarks. The recognized landmarks may be spaced apartin the sparse map at a rate of no more than 0.5 per kilometer.

In some embodiments, a system for autonomously navigating a vehiclealong a road segment based on a predetermined landmark location mayinclude at least one processor programmed to: receive from a camera, atleast one image representative of an environment of the vehicle;determine a position of the vehicle along a predetermined road modeltrajectory associated with the road segment based, at least in part, oninformation associated with the at least one image; identify arecognized landmark forward of the vehicle based on the determinedposition, wherein the recognized landmark is beyond a sight range of thecamera; determine a current distance between the vehicle and therecognized landmark by comparing the determined position of the vehiclewith a predetermined position of the recognized landmark; and determinean autonomous navigational response for the vehicle based on thedetermined current distance.

In some embodiments of the system, the predetermined position of therecognized landmark may be determined as an average of a plurality ofacquired position measurements associated with the recognized landmark,wherein the plurality of acquired position measurements are determinedbased on acquisition of at least one environmental image, analysis ofthe at least one environmental image to identify the recognized landmarkin the environment, reception of global positioning system (GPS) data,analysis of the at least one environmental image to determine a relativeposition of the recognized landmark with respect to the vehicle, anddetermination of a globally localized position of the recognizedlandmark based on at least the GPS data and the determined relativeposition. The autonomous navigational response may include applicationof brakes associated with the vehicle. The autonomous navigationalresponse may include modifying a steering angle of the vehicle. Therecognized landmark may include a stop line, a traffic light, a stopsign, or a curve along the road segment. The camera may be included inthe vehicle.

In some embodiments, a method for autonomously navigating a vehiclealong a road segment based on a predetermined landmark location mayinclude receiving from a camera, at least one image representative of anenvironment of the vehicle; determining a position of the vehicle alonga predetermined road model trajectory associated with the road segmentbased, at least in part, on information associated with the at least oneimage; identifying a recognized landmark forward of the vehicle based onthe determined position, wherein the recognized landmark is beyond asight range of the camera; determining a current distance between thevehicle and the recognized landmark by comparing the determined positionof the vehicle with a predetermined position of the recognized landmark;and determining an autonomous navigational response for the vehiclebased on the determined current distance.

In some embodiments of the method, the predetermined position of therecognized landmark may be determined as an average of a plurality ofacquired position measurements associated with the recognized landmark,wherein the plurality of acquired position measurements may bedetermined based on acquisition of at least one environmental image,analysis of the at least one environmental image to identify therecognized landmark in the environment, reception of global positioningsystem (GPS) data, analysis of the at least one environmental image todetermine a relative position of the recognized landmark with respect tothe vehicle, and determination of a globally localized position of therecognized landmark based on at least the GPS data and the determinedrelative position. The autonomous navigational response may includeapplication of brakes associated with the vehicle. The autonomousnavigational response may include modifying a steering angle of thevehicle. The recognized landmark may include a stop line, a trafficlight, a stop sign, or a curve along the road segment. The camera may beincluded in the vehicle.

In some embodiments, a non-transitory computer readable medium may storeinstructions causing at least one processor to perform autonomousnavigation of a vehicle along a road segment. The instructions may causethe processor to perform the steps of: receiving from a camera, at leastone image representative of an environment of the vehicle; determining aposition of the vehicle along a predetermined road model trajectoryassociated with the road segment based, at least in part, on informationassociated with the at least one image; identifying a recognizedlandmark forward of the vehicle based on the determined position,wherein the recognized landmark is beyond a sight range of the camera;determining a current distance between the vehicle and the recognizedlandmark by comparing the determined position of the vehicle with apredetermined position of the recognized landmark; and determining anautonomous navigational response for the vehicle based on the determinedcurrent distance.

In some embodiments of the non-transitory computer readable medium, theautonomous navigational response may include application of brakesassociated with the vehicle. The autonomous navigational response mayinclude modifying a steering angle of the vehicle. The recognizedlandmark may include a stop line, a traffic light, a stop sign, or acurve along the road segment.

In some embodiments, a system for autonomously navigating a vehiclealong a road segment may include at least one processor programmed to:receive, from at least one sensor, information relating to one or moreaspects of the road segment; determine a local feature of the roadsegment based on the received information; compare the local feature toa predetermined signature feature for the road segment; determine acurrent location of the vehicle along a predetermined road modeltrajectory associated with the road segment based on the comparison ofthe local feature and the predetermined signature feature; and determinean autonomous steering action for the vehicle based on a direction ofthe predetermined road model trajectory at the determined location.

In some embodiments of the system, the at least one processor may befurther programmed to: determine a heading direction of the vehicle atthe current location, and determine the autonomous steering action bycomparing the direction of the predetermined road model trajectory withthe heading direction. The heading direction may be determined based ona travelled trajectory of the vehicle. The at least one sensor mayinclude an image capture device configured to acquire at least one imagerepresentative of an environment of the vehicle. The signature featuremay include a road width profile over at least a portion of the roadsegment. The signature feature may include a lane width profile over atleast a portion of the road segment. The signature feature may include adashed line spacing profile over at least a portion of the road segment.The signature feature may include a predetermined number of roadmarkings along at least a portion of the road segment. The signaturefeature may include a road surface profile over at least a portion ofthe road segment. The signature feature may include a predeterminedcurvature associated with the road segment. Determining the currentlocation of the vehicle may include comparing first parameter valuesindicative of a curvature of the predetermined road model trajectory andsecond parameter values indicative of a curvature of a measuredtrajectory for the vehicle. The at least one sensor may include asuspension component monitor.

In some embodiments, a vehicle may include a body; at least one sensorconfigured to acquire information relating to one or more aspects of theroad segment; and at least one processor programmed to: determine alocal feature of the road segment based on the information received fromthe at least one sensor; compare the local feature to a predeterminedsignature feature for the road segment; determine a current location ofthe vehicle along a predetermined road model trajectory associated withthe road segment based on the comparison of the local feature and thepredetermined signature feature; and determine an autonomous steeringaction for the vehicle based on a direction of the predetermined roadmodel trajectory at the current location.

In some embodiments of the vehicle, the signature feature may include atleast one of a road width profile over at least a portion of the roadsegment, a lane width profile over at least a portion of the roadsegment, a dashed line spacing profile over at least a portion of theroad segment, a predetermined number of road markings along at least aportion of the road segment, a road surface profile over at least aportion of the road segment, and a predetermined curvature associatedwith the road segment. The vehicle may include a suspension componentmonitor, wherein the processor is further programmed to determine thelocal feature based on signals from the suspension component monitor.The processor may be further programmed to: determine a headingdirection of the vehicle; determine a direction of the predeterminedroad model trajectory at the current location; and determine theautonomous steering action by comparing the direction with the headingdirection.

In some embodiments, a method of navigating a vehicle may includereceiving, from at least one sensor, information relating to one or moreaspects of the road segment; determining, using at least one processor,a local feature of the road segment based on the information receivedfrom the at least one sensor; comparing the received information to apredetermined signature feature for the road segment; determining acurrent location of the vehicle along a predetermined road modeltrajectory associated with the road segment based on the comparison ofthe received information and the predetermined signature feature; anddetermining an autonomous steering action for the vehicle based on adirection of the predetermined road model trajectory at the currentlocation.

In some embodiments, the method may include determining a headingdirection of the vehicle at the current location; determining thedirection of the predetermined road model trajectory at the currentlocation; and determining the autonomous steering action by comparingthe direction of the predetermined road model trajectory with theheading direction. The local feature may include at least one of a roadwidth profile over at least a portion of the road segment, a lane widthprofile over at least a portion of the road segment, a dashed linespacing profile over at least a portion of the road segment, apredetermined number of road markings along at least a portion of theroad segment, a road surface profile over at least a portion of the roadsegment, and a predetermined curvature associated with the road segment.The method may include determining, using a suspension componentmonitor, a road surface profile; comparing the road surface profile witha predetermined road surface profile; and determining the currentlocation based on the comparison of the road surface profile and thepredetermined road surface profile.

In some embodiments, a system for autonomously navigating a vehicle mayinclude at least one processor programmed to: receive from a rearwardfacing camera, at least one image representing an area at a rear of thevehicle; analyze the at least one rearward facing image to locate in theimage a representation of at least one landmark; determine at least oneindicator of position of the landmark relative to the vehicle; determinea forward trajectory for the vehicle based, at least in part, upon theindicator of position of the landmark relative to the vehicle; and causethe vehicle to navigate along the determined forward trajectory.

In some embodiments of the system, the indicator of position may includea distance between the vehicle and the landmark. The indicator ofposition may include a relative angle between the vehicle and thelandmark. The landmark may include a road edge, a lane marking, areflector, a pole, a change in line pattern on a road, or a road sign.The landmark may include a backside of a road sign. The at least oneprocessor may be further programmed to determine a lane offset amount ofthe vehicle within a current lane of travel based on the indicator ofposition of the landmark, and wherein determination of the forwardtrajectory is further based on the determined lane offset amount. The atleast one processor may be further programmed to receive from anothercamera, at least one image representing another area of the vehicle, andwherein the determination of the forward trajectory is further based onthe at least one image received from the another camera.

In some embodiments, a method of autonomously navigating a vehicle mayinclude receiving from a rearward facing camera, at least one imagerepresenting an area at a rear of the vehicle;

analyzing the at least one rearward facing image to locate in the imagea representation of at least one landmark; determining at least oneindicator of position of the landmark relative to the vehicle;determining a forward trajectory for the vehicle based, at least inpart, upon the indicator of position of the landmark relative to thevehicle; and causing the vehicle to navigate along the determinedforward trajectory.

In some embodiments of the method, the indicator of position may includea distance between the vehicle and the landmark. The indicator ofposition may include a relative angle between the vehicle and thelandmark. The landmark may include a road edge, a lane marking, areflector, a pole, a change in line pattern on a road, or a road sign.The landmark may include a backside of a road sign. The method mayinclude determining a lane offset amount of the vehicle within a currentlane of travel based on the indicator of position of the landmark, andwherein the determining of the forward trajectory may be based on thedetermined lane offset amount.

In some embodiments, a vehicle may include a body; a rearward facingcamera; and at least one processor programmed to: receive, via arearward camera interface connecting the rearward facing camera, atleast one image representing an area at a rear of the vehicle; analyzethe at least one rearward facing image to locate in the image arepresentation of at least one landmark; determine at least oneindicator of position of the landmark relative to the vehicle; determinea forward trajectory for the vehicle based, at least in part, upon theindicator of position of the landmark relative to the vehicle; and

cause the vehicle to navigate along the determined forward trajectory.

In some embodiments of the vehicle, the rearward facing camera may bemounted on an object connected to the vehicle. The object may be atrailer, a bike carrier, a ski/snowboard carrier, a mounting base, or aluggage carrier. The rearward camera interface may include a detachableinterface. The rearward camera interface may include a wirelessinterface.

In some embodiments, a system for navigating a vehicle by determining afree space region in which a vehicle can travel may include at least oneprocessor programmed to: receive from an image capture device, aplurality of images associated with an environment of a vehicle; analyzeat least one of the plurality of images to identify a first free spaceboundary on a driver side of the vehicle and extending forward of thevehicle, a second free space boundary on a passenger side of the vehicleand extending forward of the vehicle, and a forward free space boundaryforward of the vehicle and extending between the first free spaceboundary and the second free space boundary; wherein the first freespace boundary, the second free space boundary, and the forward freespace boundary define a free space region forward of the vehicle;determine a navigational path for the vehicle through the free spaceregion; and cause the vehicle to travel on at least a portion of thedetermined navigational path within the free space region forward of thevehicle.

In some embodiments of the system, the first free space boundary maycorrespond to at least one of a road edge, a curb, a barrier, a lanedividing structure, a parked vehicle, a tunnel wall, or a bridgestructure. The second free space boundary may correspond to at least oneof a road edge, a curb, a barrier, a lane dividing structure, a parkedvehicle, a tunnel wall, or a bridge structure. The forward free spaceboundary may correspond to a road horizon line. The at least oneprocessor may be further programmed to identify, based on analysis ofthe at least one of the plurality of images, an obstacle forward of thevehicle and exclude the identified obstacle from the free space regionforward of the vehicle. The obstacle may include a pedestrian. Theobstacle may include another vehicle. The obstacle may include debris.The at least one processor may be further programmed to identify, basedon analysis of the at least one of the plurality of images, an obstacleforward of the vehicle and exclude a region surrounding the identifiedobstacle from the free space region forward of the vehicle. The at leastone processor may be further programmed to determine the regionsurrounding the identified obstacle based on one or more of thefollowing: a speed of the vehicle, a type of the obstacle, an imagecapture rate of the image capture device, and a movement speed of theobstacle.

In some embodiments, a vehicle may include a body, the body including adriver side and a passenger side; an image capture device; and at leastone processor programmed to: receive from the image capture device, aplurality of images associated with an environment of the vehicle;analyze at least one of the plurality of images to identify a first freespace boundary on the driver side of the body and extending forward ofthe body, a second free space boundary on the passenger side of the bodyand extending forward of the body, and a forward free space boundaryforward of the body and extending between the first free space boundaryand the second free space boundary; wherein the first free spaceboundary, the second free space boundary, and the forward free spaceboundary define a free space region forward of the body; determine anavigational path for the vehicle through the free space region; andcause the vehicle to travel on at least a portion of the determinednavigational path within the free space region forward of the vehicle.

In some embodiments, a method of navigating a vehicle by determining afree space region in which a vehicle can travel may include receivingfrom an image capture device, a plurality of images associated with anenvironment of a vehicle; analyzing at least one of the plurality ofimages to identify a first free space boundary on a driver side of thevehicle and extending forward of the vehicle, a second free spaceboundary on a passenger side of the vehicle and extending forward of thevehicle, and a forward free space boundary forward of the vehicle andextending between the first free space boundary and the second freespace boundary; wherein the first free space boundary, the second freespace boundary, and the forward free space boundary define a free spaceregion forward of the vehicle; determining a navigational path for thevehicle through the free space region; and causing the vehicle to travelon at least a portion of the determined navigational path within thefree space region forward of the vehicle.

In some embodiments of the method, the first free space boundary maycorrespond to at least one of a road edge, a curb, a barrier, a lanedividing structure, a parked vehicle, a tunnel wall, or a bridgestructure. The second free space boundary may correspond to at least oneof a road edge, a curb, a barrier, a lane dividing structure, a parkedvehicle, a tunnel wall, or a bridge structure. The forward free spaceboundary may correspond to a road horizon line. The method may includeidentifying, based on analysis of the at least one of the plurality ofimages, an obstacle forward of the vehicle; and excluding the identifiedobstacle from the free space region forward of the vehicle. The obstaclemay include a pedestrian. The obstacle may include another vehicle. Theobstacle may include debris. The method may include identifying, basedon analysis of the at least one of the plurality of images, an obstacleforward of the vehicle; and excluding a region surrounding theidentified obstacle from the free space region forward of the vehicle.The method may include determining the region surrounding the identifiedobstacle based on one or more of the following: a speed of the vehicle,a type of the obstacle, an image capture rate of the image capturedevice, and a movement speed of the obstacle.

In some embodiments, a system for navigating a vehicle on a road withsnow covering at least some lane markings and road edges may include atleast one processor programmed to: receive from an image capture device,at least one environmental image forward of the vehicle, including areaswhere snow covers at least some lane markings and road edges; identify,based on an analysis of the at least one image, at least a portion ofthe road that is covered with snow and probable locations for road edgesbounding the at least a portion of the road that is covered with snow;and cause the vehicle to navigate a navigational path that includes theidentified portion of the road and falls within the determined probablelocations for the road edges.

In some embodiments of the system, the analysis of the at least oneimage may include identifying at least one tire track in the snow. Theanalysis of the at least one image may include identifying a change oflight across a surface of the snow. The analysis of the at least oneimage may include identifying a plurality of trees along an edge of theroad. The analysis of the at least one image may include recognizing achange in curvature at a surface of the snow. The recognized change incurvature may be determined to correspond to a probable location of aroad edge. The analysis of the at least one image may include a pixelanalysis of the at least one image in which at least a first pixel iscompared to at least a second pixel in order to determine a featureassociated with a surface of the snow covering at least some lanemarkings and road edges. The feature may correspond to an edge of a tiretrack. The feature may correspond to an edge of the road. The at leastone processor may be further programmed to cause the vehicle to navigatebetween determined edges of the road. The at least one processor may befurther programmed to cause the vehicle to navigate by at leastpartially following tire tracks in the snow.

In some embodiments, a method of navigating a vehicle on a road withsnow covering at least some lane markings and road edges may includereceiving from an image capture device, at least one environmental imageforward of the vehicle, including areas where snow covers at least somelane markings and road edges; identifying, based on an analysis of theat least one image, at least a portion of the road that is covered withsnow and probable locations for road edges bounding the at least aportion of the road that is covered with snow; and causing the vehicleto navigate a navigational path that includes the identified portion ofthe road and falls within the determined probable locations for the roadedges.

In some embodiments of the method, the analysis of the at least oneimage may include identifying at least one tire track in the snow. Theanalysis of the at least one image may include identifying a change oflight across a surface of the snow. The analysis of the at least oneimage may include identifying a plurality of trees along an edge of theroad. The analysis of the at least one image may include recognizing achange in curvature at a surface of the snow. The recognized change incurvature may be determined to correspond to a probable location of aroad edge. The analysis of the at least one image may include a pixelanalysis of the at least one image in which at least a first pixel iscompared to at least a second pixel in order to determine a featureassociated with a surface of the snow covering at least some lanemarkings and road edges. The feature may correspond to an edge of a tiretrack. The feature may correspond to an edge of the road. The method mayinclude causing the vehicle to navigate between determined edges of theroad. The method may include causing the vehicle to navigate by at leastpartially following tire tracks in the snow.

In some embodiments, a system for navigating a vehicle on a road atleast partially covered with snow may include at least one processorprogrammed to: receive from an image capture device, a plurality ofimages captured of an environment forward of the vehicle, includingareas where snow covers a road on which the vehicle travels; analyze atleast one of the plurality of images to identify a first free spaceboundary on a driver side of the vehicle and extending forward of thevehicle, a second free space boundary on a passenger side of the vehicleand extending forward of the vehicle, and a forward free space boundaryforward of the vehicle and extending between the first free spaceboundary and the second free space boundary; wherein the first freespace boundary, the second free space boundary, and the forward freespace boundary define a free space region forward of the vehicle;determine a first proposed navigational path for the vehicle through thefree space region; provide the at least one of the plurality of imagesto a neural network and receive from the neural network a secondproposed navigational path for the vehicle based on analysis of the atleast one of the plurality of images by the neural network; determinewhether the first proposed navigational path agrees with the secondproposed navigational path; and cause the vehicle to travel on at leasta portion of the first proposed navigational path if the first proposednavigational path is determined to agree with the second proposednavigational path.

In some embodiments, a system for calibrating an indicator of speed ofan autonomous vehicle may include at least one processor programmed to:receive from a camera a plurality of images representative of anenvironment of the vehicle; analyze the plurality of images to identifyat least two recognized landmarks; determine, based on known locationsof the two recognized landmarks, a value indicative of a distancebetween the at least two recognized landmarks; determine, based on anoutput of at least one sensor associated with the autonomous vehicle, ameasured distance between the at least two landmarks; and determine acorrection factor for the at least one sensor based on a comparison ofthe value indicative of the distance between the at least to recognizedlandmarks and the measured distance between the at least two landmarks.

In some embodiments of the system, the correction factor may bedetermined such that an operation on the determined distance along theroad segment by the correction factor matches the distance valuereceived via the wireless transceiver. The two recognized landmarks mayinclude one or more of a traffic sign, an arrow marking, a lane marking,a dashed lane marking, a traffic light, a stop line, a directional sign,a reflector, a landmark beacon, or a lamppost. The at least one sensormay include a speedometer associated with the vehicle. The knownlocations of the two recognized landmarks may be received from a serverbased system located remotely with respect to the vehicle. Each of theknown locations may constitute a refined location determined based on aplurality of GPS-based measurements.

In some embodiments, a system for calibrating an indicator of speed ofan autonomous vehicle may include at least one processor programmed to:determine a distance along a road segment based on an output of at leastone sensor associated with the autonomous vehicle; receive, via awireless transceiver, a distance value associated with the road segment;and determine a correction factor for the at least one sensor based onthe determined distance along the road segment and the distance valuereceived via the wireless transceiver.

In some embodiments of the system, the distance value associated withthe road segment, received via the wireless transceiver, may bedetermined based on prior measurements made by a plurality of measuringvehicles. The plurality of measuring vehicles may include at least 100measuring vehicles. The plurality of measuring vehicles may include atleast 1000 measuring vehicles. The correction factor may be determinedsuch that an operation on the determined distance along the road segmentby the correction factor matches the distance value received via thewireless transceiver. The at least one processor may be programmed todetermine a composite correction factor based on a plurality ofdetermined correction factors. The composite correction factor may bedetermined by averaging the plurality of determined correction factors.The composite correction factor may be determined by finding a mean ofthe plurality of determined correction factors.

In some embodiments, a vehicle may include a body; a camera; and atleast one processor programmed to: receive from the camera a pluralityof images representative of an environment of the vehicle; analyze theplurality of images to identify at least two recognized landmarks;determine, based on known locations of the two recognized landmarks, avalue indicative of a distance between the at least two recognizedlandmarks; determine, based on an output of at least one sensorassociated with the autonomous vehicle, a measured distance between theat least two landmarks; and determine a correction factor for the atleast one sensor based on a comparison of the value indicative of thedistance between the at least to recognized landmarks and the measureddistance between the at least two landmarks.

In some embodiments of the vehicle, the at least one sensor may includea speedometer associated with the vehicle. The two recognized landmarksmay include one or more of a traffic sign, an arrow marking, a lanemarking, a dashed lane marking, a traffic light, a stop line, adirectional sign, a reflector, a landmark beacon, or a lamppost. Theknown locations of the two recognized landmarks may be received from aserver based system located remotely with respect to the vehicle.

In some embodiments, a system for determining a lane assignment for anautonomous vehicle along a road segment may include at least oneprocessor programmed to: receive from a camera at least one imagerepresentative of an environment of the vehicle; analyze the at leastone image to identify at least one recognized landmark; determine anindicator of a lateral offset distance between the vehicle and the atleast one recognized landmark; and determine a lane assignment of thevehicle along the road segment based on the indicator of the lateraloffset distance between the vehicle and the at least one recognizedlandmark.

In some embodiments of the system, the environment of the vehicle mayinclude the road segment, a number of lanes, and the at least onerecognized landmark. The at least one recognized landmark may include atleast one of a traffic sign, an arrow marking, a lane marking, a dashedlane marking, a traffic light, a stop line, a directional sign, areflector, a landmark beacon, or a lamppost. The at least one recognizedlandmark may include a sign for a business. The lateral offset distancebetween the vehicle and the at least one recognized landmark may be asum of a first distance between the vehicle and a first side of the roadsegment and a second distance between the first side of the road and theat least one recognized landmark. The determination of the indicator ofthe lateral offset distance between the vehicle and the at least onerecognized landmark may be based on a predetermined position of the atleast one recognized landmark. The determination of the indicator of thelateral offset distance between the vehicle and the at least onerecognized landmark may be based on a scale associated with the at leastone image. The determination of the lane assignment may be further basedon at least one of a width of the road segment, a number of lanes of theroad segment, and a lane width. The determination of the lane assignmentmay be further based on a predetermined road model trajectory associatedwith the road segment. The at least one recognized landmark may includea first recognized landmark on a first side of the vehicle and a secondrecognized landmark on a second side of the vehicle and whereindetermination of the lane assignment of the vehicle along the roadsegment is based on a first indicator of lateral offset distance betweenthe vehicle and the first recognized landmark and a second indicator oflateral offset distance between the vehicle and the second recognizedlandmark.

In some embodiments, a computer-implemented method for determining alane assignment for an autonomous vehicle along a road segment mayinclude the following operations performed by one or more processors:receiving from a camera at least one image representative of anenvironment of the vehicle; analyzing the at least one image to identifyat least one recognized landmark; determining an indicator of a lateraloffset distance between the vehicle and the at least one recognizedlandmark; and determining a lane assignment of the vehicle along theroad segment based on the indicator of the lateral offset distancebetween the vehicle and the at least one recognized landmark.

In some embodiments of the method, the at least one recognized landmarkmay include at least one of a traffic sign, an arrow marking, a lanemarking, a dashed lane marking, a traffic light, a stop line, adirectional sign, a reflector, a landmark beacon, or a lamppost. The atleast one recognized landmark may include a sign for a business. Thedetermination of the lane assignment may be further based on apredetermined road model trajectory associated with the road segment.The at least one recognized landmark may include a first recognizedlandmark on a first side of the vehicle and a second recognized landmarkon a second side of the vehicle and wherein determination of the laneassignment of the vehicle along the road segment is based on a firstindicator of lateral offset distance between the vehicle and the firstrecognized landmark and a second indicator of lateral offset distancebetween the vehicle and the second recognized landmark.

In some embodiments, a computer-readable storage medium may include aset of instructions that are executable by at least one processor tocause the at least one processor to perform a method for determining alane assignment for an autonomous vehicle along a road segment. Themethod may include receiving from a camera at least one imagerepresentative of an environment of the vehicle; analyzing the at leastone image to identify at least one recognized landmark; determining anindicator of a lateral offset distance between the vehicle and the atleast one recognized landmark; and determining a lane assignment of thevehicle along the road segment based on the indicator of the lateraloffset distance between the vehicle and the at least one recognizedlandmark.

In some embodiments of the computer-readable storage medium, the atleast one recognized landmark may include at least one of a trafficsign, an arrow marking, a lane marking, a dashed lane marking, a trafficlight, a stop line, a directional sign, a reflector, a landmark beacon,or a lamppost. The at least one recognized landmark may include a signfor a business. The determination of the lane assignment may be furtherbased on a predetermined road model trajectory associated with the roadsegment. The at least one recognized landmark may include a firstrecognized landmark on a first side of the vehicle and a secondrecognized landmark on a second side of the vehicle and whereindetermination of the lane assignment of the vehicle along the roadsegment is based on a first indicator of lateral offset distance betweenthe vehicle and the first recognized landmark and a second indicator oflateral offset distance between the vehicle and the second recognizedlandmark.

In some embodiments, a system for autonomously navigating a vehiclealong a road segment may include at least one processor programmed to:receive from a camera at least one image representative of anenvironment of the vehicle; analyze the at least one image to identifyat least one recognized landmark, wherein the at least one recognizedlandmark is part of a group of recognized landmarks, and identificationof the at least one recognized landmark is based, at least in part, uponone or more landmark group characteristics associated with the group ofrecognized landmarks; determine a current location of the vehiclerelative to a predetermined road model trajectory associated with theroad segment based, at least in part, on a predetermined location of therecognized landmark; and determine an autonomous steering action for thevehicle based on a direction of the predetermined road model trajectoryat the determined current location of the vehicle relative to thepredetermined road model trajectory.

In some embodiments of the system, the at least one recognized landmarkmay include at least one of a traffic sign, an arrow marking, a lanemarking, a dashed lane marking, a traffic light, a stop line, adirectional sign, a reflector, a landmark beacon, or a lamppost. The atleast one recognized landmark may include a sign for a business. Thepredetermined road model trajectory may include a three-dimensionalpolynomial representation of a target trajectory along the road segment.The at least one processor may be further programmed to determine acurrent location of the vehicle along the predetermined road modeltrajectory based on a vehicle velocity. The one or more landmark groupcharacteristics may include relative distances between members of thegroup of recognized landmarks. The one or more landmark groupcharacteristics may include an ordering sequence of members of the groupof recognized landmarks. The one or more landmark group characteristicsmay include a number of landmarks included in the group of recognizedlandmarks. Identification of the at least one recognized landmark may bebased, at least in part, upon a super landmark signature associated withthe group of recognized landmarks. The at least one processor may beprogrammed to determine an autonomous steering action for the vehicle bycomparing a heading direction of the vehicle to the predetermined roadmodel trajectory at the determined current location of the vehicle.

In some embodiments, a computer-implemented method for autonomouslynavigating a vehicle along a road segment may include the followingoperations performed by one or more processors: receiving from a cameraat least one image representative of an environment of the vehicle;analyzing the at least one image to identify at least one recognizedlandmark, wherein the at least one recognized landmark is part of agroup of recognized landmarks, and identification of the at least onerecognized landmark is based, at least in part, upon one or morelandmark group characteristics associated with the group of recognizedlandmarks; deter mining, relative to the vehicle, a current location ofthe vehicle relative to a predetermined road model trajectory associatedwith the road segment based, at least in part, on a predeterminedlocation of the recognized landmark; and determining an autonomoussteering action for the vehicle based on a direction of thepredetermined road model trajectory at the determined current locationof the vehicle relative to the predetermined road model trajectory.

In some embodiments of the method, the at least one recognized landmarkmay include at least one of a traffic sign, an arrow marking, a lanemarking, a dashed lane marking, a traffic light, a stop line, adirectional sign, a reflector, a landmark beacon, or a lamppost. The atleast one recognized landmark may include a sign for a business. The oneor more landmark group characteristics may include relative distancesbetween members of the group of recognized landmarks. The one or morelandmark group characteristics may include an ordering sequence ofmembers of the group of recognized landmarks.

In some embodiments, a computer-readable storage medium may include aset of instructions that are executable by at least one processor tocause the at least one processor to perform a method for autonomouslynavigating a vehicle along a road segment. The method may includereceiving from a camera at least one image representative of anenvironment of the vehicle; analyzing the at least one image to identifyat least one recognized landmark, wherein the at least one recognizedlandmark is part of a group of recognized landmarks, and identificationof the at least one recognized landmark is based, at least in part, uponone or more landmark group characteristics associated with the group ofrecognized landmarks; determining, relative to the vehicle, a currentlocation of the vehicle relative to a predetermined road modeltrajectory associated with the road segment based, at least in part, ona predetermined location of the recognized landmark; and determining anautonomous steering action for the vehicle based on a direction of thepredetermined road model trajectory at the determined current locationof the vehicle relative to the predetermined road model trajectory.

In some embodiments of the computer-readable storage medium, the atleast one landmark may include at least one of a traffic sign, an arrowmarking, a lane marking, a dashed lane marking, a traffic light, a stopline, a directional sign, a reflector, a landmark beacon, or a lamppost.The at least one recognized landmark may include a sign for a business.The one or more landmark group characteristics may include relativedistances between members of the group of recognized landmarks. The oneor more landmark group characteristics may include an ordering sequenceof members of the group of recognized landmarks.

In some embodiments, a navigation system for a vehicle may include atleast one processor programmed to: receive from a camera, at least oneenvironmental image associated with the vehicle; determine anavigational maneuver for the vehicle based on analysis of the at leastone environmental image; cause the vehicle to initiate the navigationalmaneuver; receive a user input, associated with a user's navigationalresponse different from the initiated navigational maneuver; determinenavigational situation information relating to the vehicle based on thereceived user input; and store the navigational situation information inassociation with information relating to the user input.

In some embodiments of the system, the navigational maneuver may bebased on a recognized landmark identified in the at least oneenvironmental image. The information relating to the user input mayinclude information specifying at least one of a degree of a turn of thevehicle, an amount of an acceleration of the vehicle, and an amount ofbraking of the vehicle. The control system may include at least one of asteering control, an acceleration control, and a braking control. Thenavigational situation information may include one or more imagescaptured by a camera onboard the vehicle. The user input may include atleast one of braking, steering, or accelerating. The navigationalsituation information may include a location of the vehicle. Thenavigational situation information may include at least one output of asensor onboard the vehicle. The sensor may be a speedometer. The sensormay be an accelerometer. The sensor may be an IR sensor. Thenavigational situation information may include a time of day. Thenavigational situation information may include an indication of thepresence of a vision inhibitor. The vision inhibitor may be caused byglare. The navigational situation information may be determined based onthe at least one environmental image. The system may include atransmitter for sending the navigational situation information to aserver remote from the vehicle.

In some embodiments, a non-transitory computer-readable medium mayinclude instructions that are executable by at least one processor tocause the at least one processor to perform a method. The method mayinclude receiving from a camera, at least one environmental imageassociated with the vehicle; determining a navigational maneuver for thevehicle based on analysis of the at least one environmental image;causing the vehicle to initiate the navigational maneuver; receiving auser input, associated with a user's navigational response differentfrom the initiated navigational maneuver; determining navigationalsituation information relating to the vehicle based on the received userinput; and storing the navigational situation information in associationwith information relating to the user input.

In some embodiments, a navigation system for a vehicle may include atleast one processor programmed to: determine a navigational maneuver forthe vehicle based, at least in part, on a comparison of a motion of thevehicle with respect to a predetermined model representative of a roadsegment; receive from a camera, at least one image representative of anenvironment of the vehicle; determine, based on analysis of the at leastone image, an existence in the environment of the vehicle of anavigational adjustment condition; cause the vehicle to adjust thenavigational maneuver based on the existence of the navigationaladjustment condition; and store information relating to the navigationaladjustment condition.

In some embodiments of the system, the navigational adjustment conditionmay include a parked car. The navigational adjustment condition mayinclude a lane shift. The navigational adjustment condition may includeat least one of a newly encountered traffic sign or a newly encounteredtraffic light. The navigational adjustment condition may include an areaof construction. The processor may be further programmed to cause thestored information relating to the navigational adjustment condition tobe transmitted to a road model management system for determining whetheran update to the predetermined model representative of the road segmentis warranted by the navigational adjustment condition. The informationstored relative to the navigational adjustment condition may include atleast one of an indicator of location where the navigational adjustmentcondition was encountered, an indication of the adjustment made to thenavigational maneuver, and the at least one image. The predeterminedmodel representative of the road segment may include a three-dimensionalspline representing a predetermined path of travel along the roadsegment.

In some embodiments, a method for navigating a vehicle may includedetermining a navigational maneuver for the vehicle based, at least inpart, on a comparison of a motion of the vehicle with respect to apredetermined model representative of a road segment; receiving from acamera, at least one image representative of an environment of thevehicle; determining, based on analysis of the at least one image, anexistence in the environment of the vehicle of a navigational adjustmentcondition; causing the vehicle to adjust the navigational maneuver basedon the existence of the navigational adjustment condition; and storinginformation relating to the navigational adjustment condition.

In some embodiments of the method, the navigational adjustment conditionmay include a parked ear. The navigational adjustment condition mayinclude a lane shift. The navigational adjustment condition may includeat least one of a newly encountered traffic sign or a newly encounteredtraffic light. The navigational adjustment condition may include an areaof construction. The method may include causing the stored informationrelating to the navigational adjustment condition to be transmitted to aroad model management system for determining whether an update to thepredetermined model representative of the road segment is warranted bythe navigational adjustment condition. The information stored relativeto the navigational adjustment condition may include at least one of anindicator of location where the navigational adjustment condition wasencountered, an indication of the adjustment made to the navigationalmaneuver, and the at least one image. The predetermined modelrepresentative of the road segment may include a three-dimensionalspline representing a predetermined path of travel along the roadsegment.

In some embodiments, a non-transitory computer-readable medium mayinclude instructions that are executable by at least one processor tocause the at least one processor to perform a method. The method mayinclude determining a navigational maneuver for the vehicle based, atleast in part, on a comparison of a motion of the vehicle with respectto a predetermined model representative of a road segment; receivingfrom a camera, at least one image representative of an environment ofthe vehicle; determining, based on analysis of the at least one image,an existence in the environment of the vehicle of a navigationaladjustment condition; causing the vehicle to adjust the navigationalmaneuver based on the existence of the navigational adjustmentcondition; and storing information relating to the navigationaladjustment condition.

In some embodiments of the computer-readable medium, the method mayinclude causing the stored information relating to the navigationaladjustment condition to be transmitted to a road model management systemfor determining whether an update to the predetermined modelrepresentative of the road segment is warranted by the navigationaladjustment condition. The information stored relative to thenavigational adjustment condition may include at least one of anindicator of location where the navigational adjustment condition wasencountered, an indication of the adjustment made to the navigationalmaneuver, and the at least one image. The predetermined modelrepresentative of the road segment may include a three-dimensionalspline representing a predetermined path of travel along the roadsegment.

In some embodiments, a system for interacting with a plurality ofautonomous vehicles may include a memory including a predetermined modelrepresentative of at least one road segment; and at least one processorprogrammed to: receive from each of the plurality of autonomous vehiclesnavigational situation information associated with an occurrence of anadjustment to a determined navigational maneuver; analyze thenavigational situation information; determine, based on the analysis ofthe navigational situation information, whether the adjustment to thedetermined navigational maneuver was due to a transient condition; andupdate the predetermined model representative of the at least one roadsegment if the adjustment to the determined navigational maneuver wasnot due to a transient condition.

In some embodiments of the system, the predetermined modelrepresentative of at least one road segment may include athree-dimensional spline representing a predetermined path of travelalong the at least one road segment. The update to the predeterminedmodel may include an update to the three-dimensional spline representinga predetermined path of travel along the at least one road segment. Theadjustment to a determined navigational maneuver may be resulted from auser intervention. The adjustment to a determined navigational maneuvermay be resulted from an automatic determination, based on imageanalysis, of an existence in a vehicle environment of a navigationaladjustment condition. The navigational situation information may includeat least one image representing an environment of an autonomous vehicle.The navigational situation information may include a video representingan environment of an autonomous vehicle. The transient condition may beassociated with a parked car, an intervening car, a pedestrian, a lowlight condition, a glare condition, a temporary barrier, or temporaryroadwork.

In some embodiments, a method for interacting with a plurality ofautonomous vehicles may include receiving from each of the plurality ofautonomous vehicles navigational situation information associated withan occurrence of an adjustment to a determined navigational maneuver;analyzing the navigational situation information; determining, based onthe analysis of the navigational situation information, whether theadjustment to the determined navigational maneuver was due to atransient condition; and updating a predetermined model representativeof the at least one road segment if the adjustment to the determinednavigational maneuver was not due to a transient condition.

In some embodiments of the method, the predetermined modelrepresentative of at least one road segment may include athree-dimensional spline representing a predetermined path of travelalong the at least one road segment. The update to the predeterminedmodel may include an update to the three-dimensional spline representinga predetermined path of travel along the at least one road segment. Theadjustment to a determined navigational maneuver may be resulted from auser intervention. The adjustment to a determined navigational maneuvermay be resulted from an automatic determination, based on imageanalysis, of an existence in a vehicle environment of a navigationaladjustment condition. The navigational situation information may includeat least one image representing an environment of an autonomous vehicle.The navigational situation information may include a video representingan environment of an autonomous vehicle. The transient condition may beassociated with a parked car, an intervening car, a pedestrian, a lowlight condition, a glare condition, a temporary barrier, or temporaryroadwork.

In some embodiments, a non-transitory computer-readable medium mayinclude instructions that are executable by at least one processor tocause the at least one processor to perform a method. The method mayinclude receiving from each of the plurality of autonomous vehiclesnavigational situation information associated with an occurrence of anadjustment to a determined navigational maneuver; analyzing thenavigational situation information; determining, based on the analysisof the navigational situation information, whether the adjustment to thedetermined navigational maneuver was due to a transient condition; andupdating the predetermined model representative of the at least one roadsegment if the adjustment to the determined navigational maneuver wasnot due to a transient condition.

In some embodiments of the computer-readable medium, the predeterminedmodel representative of at least one road segment may include athree-dimensional spline representing a predetermined path of travelalong the at least one road segment. Updating the predetermined modelmay include an update to the three-dimensional spline representing apredetermined path of travel along the at least one road segment. Thetransient condition may be associated with a parked car, an interveningcar, a pedestrian, a low light condition, a glare condition, a temporarybarrier, or temporary roadwork.

In some embodiments, a system for interacting with a plurality ofautonomous vehicles may include a memory including a predetermined roadmodel representative of at least one road segment; and at least oneprocessor programmed to: selectively receive, from the plurality ofautonomous vehicles, road environment information based on navigation bythe plurality of autonomous vehicles through their respective roadenvironments; determine whether one or more updates to the predeterminedroad model are required based on the road environment information; andupdate the predetermined road model to include the one or more updates.

In some embodiments of the system, the road model may include athree-dimensional spline representing a predetermined path of travelalong the at least one road segment. Selectively receiving the roadenvironment information may include a limitation on a frequency ofinformation transmissions received from a particular vehicle.Selectively receiving the road environment information may include alimitation on a frequency of information transmissions received from agroup of vehicles. Selectively receiving the road environmentinformation may include a limitation on a frequency of informationtransmissions received from vehicles traveling within a particulargeographic region. Selectively receiving the road environmentinformation may include a limitation on a frequency of informationtransmissions received from vehicles based on a determined modelconfidence level associated with a particular geographic region.Selectively receiving the road environment information may include alimitation on information transmissions received from vehicles to onlythose transmissions that include a potential discrepancy with respect toat least one aspect of the predetermined road model.

In some embodiments, a method for interacting with a plurality ofautonomous vehicles may include selectively receiving, from theplurality of autonomous vehicles, road environment information based onnavigation by the plurality of autonomous vehicles through theirrespective road environments; determining whether one or more updates tothe predetermined road model are required based on the road environmentinformation; and updating the predetermined road model to include theone or more updates.

In some embodiments of the method, the road model may include athree-dimensional spline representing a predetermined path of travelalong the at least one road segment. Selectively receiving the roadenvironment information may include a limitation on a frequency ofinformation transmissions received from a particular vehicle.Selectively receiving the road environment information may include alimitation on a frequency of information transmissions received from agroup of vehicles. Selectively receiving the road environmentinformation may include a limitation on a frequency of informationtransmissions received from vehicles traveling within a particulargeographic region. Selectively receiving the road environmentinformation may include a limitation on a frequency of informationtransmissions received from vehicles based on a determined modelconfidence level associated with a particular geographic region.Selectively receiving the road environment information may include alimitation on information transmissions received from vehicles to onlythose transmissions that include a potential discrepancy with respect toat least one aspect of the predetermined road model.

In some embodiments, a non-transitory computer-readable medium mayinclude instructions that are executable by at least one processor tocause the at least one processor to perform a method. The method mayinclude selectively receiving, from the plurality of autonomousvehicles, road environment information based on navigation by theplurality of autonomous vehicles through their respective roadenvironments; determining whether one or more updates to thepredetermined road model are required based on the road environmentinformation; and updating the predetermined road model to include theone or more updates.

In some embodiments of the computer-readable medium, the road model mayinclude a three-dimensional spline representing a predetermined path oftravel along the at least one road segment. Selectively receiving theroad environment information may include a limitation on a frequency ofinformation transmissions received from a particular vehicle.Selectively receiving the road environment information may include alimitation on a frequency of information transmissions received from agroup of vehicles. Selectively receiving the road environmentinformation may include a limitation on a frequency of informationtransmissions received from vehicles traveling within a particulargeographic region. Selectively receiving the road environmentinformation may include a limitation on information transmissionsreceived from vehicles to only those transmissions that include apotential discrepancy with respect to at least one aspect of thepredetermined road model.

Consistent with other disclosed embodiments, non-transitorycomputer-readable storage media may store program instructions, whichare executed by at least one processing device and perform any of themethods described herein.

The foregoing general description and the following detailed descriptionare exemplary and explanatory only and are not restrictive of theclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of this disclosure, illustrate various disclosed embodiments. Inthe drawings:

FIG. 1 is a diagrammatic representation of an exemplary systemconsistent with the disclosed embodiments.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments.

FIG. 2B is a diagrammatic top view representation of the vehicle andsystem shown in FIG. 2A consistent with the disclosed embodiments.

FIG. 2C is a diagrammatic top view representation of another embodimentof a vehicle including a system consistent with the disclosedembodiments.

FIG. 2D is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2E is a diagrammatic top view representation of yet anotherembodiment of a vehicle including a system consistent with the disclosedembodiments.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems consistent with the disclosed embodiments.

FIG. 3A is a diagrammatic representation of an interior of a vehicleincluding a rearview mirror and a user interface for a vehicle imagingsystem consistent with the disclosed embodiments.

FIG. 3B is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 3C is an illustration of the camera mount shown in FIG. 3B from adifferent perspective consistent with the disclosed embodiments.

FIG. 3D is an illustration of an example of a camera mount that isconfigured to be positioned behind a rearview mirror and against avehicle windshield consistent with the disclosed embodiments.

FIG. 4 is an exemplary block diagram of a memory configured to storeinstructions for performing one or more operations consistent with thedisclosed embodiments.

FIG. 5A is a flowchart showing an exemplary process for causing one ormore navigational responses based on monocular image analysis consistentwith disclosed embodiments.

FIG. 5B is a flowchart showing an exemplary process for detecting one ormore vehicles and/or pedestrians in a set of images consistent with thedisclosed embodiments.

FIG. 5C is a flowchart showing an exemplary process for detecting roadmarks and/or lane geometry information in a set of images consistentwith the disclosed embodiments.

FIG. 5D is a flowchart showing an exemplary process for detectingtraffic lights in a set of images consistent with the disclosedembodiments.

FIG. 5E is a flowchart showing an exemplary process for causing one ormore navigational responses based on a vehicle path consistent with thedisclosed embodiments.

FIG. 5F is a flowchart showing an exemplary process for determiningwhether a leading vehicle is changing lanes consistent with thedisclosed embodiments.

FIG. 6 is a flowchart showing an exemplary process for causing one ormore navigational responses based on stereo image analysis consistentwith the disclosed embodiments.

FIG. 7 is a flowchart showing an exemplary process for causing one ormore navigational responses based on an analysis of three sets of imagesconsistent with the disclosed embodiments.

FIG. 8 shows a sparse map for providing autonomous vehicle navigation,consistent with the disclosed embodiments.

FIG. 9A illustrates a polynomial representation of a portions of a roadsegment consistent with the disclosed embodiments.

FIG. 9B illustrates a curve in three-dimensional space representing atarget trajectory of a vehicle, for a particular road segment, includedin a sparse map consistent with the disclosed embodiments.

FIG. 10 illustrates example landmarks that may be included in sparse mapconsistent with the disclosed embodiments.

FIG. 11A shows polynomial representations of trajectories consistentwith the disclosed embodiments.

FIGS. 11B and 11C show target trajectories along a multi-lane roadconsistent with disclosed embodiments.

FIG. 11D shows an example road signature profile consistent withdisclosed embodiments.

FIG. 12 is a schematic illustration of a system that uses crowd sourcingdata received from a plurality of vehicles for autonomous vehiclenavigation, consistent with the disclosed embodiments.

FIG. 13 illustrates an example autonomous vehicle road navigation modelrepresented by a plurality of three dimensional splines, consistent withthe disclosed embodiments.

FIG. 14 illustrates a block diagram of a server consistent with thedisclosed embodiments.

FIG. 15 illustrates a block diagram of a memory consistent with thedisclosed embodiments.

FIG. 16 illustrates a process of clustering vehicle trajectoriesassociated with vehicles, consistent with the disclosed embodiments.

FIG. 17 illustrates a navigation system for a vehicle, which may be usedfor autonomous navigation, consistent with the disclosed embodiments.

FIG. 18 is a flowchart showing an example process for processing vehiclenavigation information for use in autonomous vehicle navigation,consistent with the disclosed embodiments.

FIG. 19 is a flowchart showing an example process performed by anavigation system of a vehicle, consistent with the disclosedembodiments.

FIG. 20 shows an example diagram of a memory consistent with thedisclosed embodiments.

FIG. 21 is a flowchart illustrating an example process for uploading arecommended trajectory to a server consistent with the disclosedembodiments.

FIG. 22 illustrates an example environment including a system foridentifying a landmark for use in autonomous vehicle navigationconsistent with the disclosed embodiments.

FIG. 23 illustrates an example environment including a system foridentifying a landmark for use in autonomous vehicle navigationconsistent with the disclosed embodiments.

FIG. 24 illustrates a method of determining a condensed signaturerepresentation of a landmark consistent with the disclosed embodiments.

FIG. 25 illustrates another method of determining a condensed signaturerepresentation of a landmark consistent with the disclosed embodiments.

FIG. 26 illustrates an example block diagram of a memory consistent withthe disclosed embodiments.

FIG. 27 is a flowchart showing an exemplary process for determining anidentifier of a landmark consistent with the disclosed embodiments.

FIG. 28 is a flowchart showing an exemplary process for updating anddistributing a vehicle road navigation model based on an identifierconsistent with the disclosed embodiments.

FIG. 29 illustrates an example block diagram of a system for determininga location of a landmark for use in navigation of an autonomous vehicleconsistent with the disclosed embodiments.

FIG. 30 illustrates an example block diagram of a memory consistent withthe disclosed embodiments.

FIG. 31 illustrates an example scaling method for determining a distancefrom a vehicle to a landmark consistent with the disclosed embodiments.

FIG. 32 illustrates an example optical flow method for determining adistance from a vehicle to a landmark consistent with the disclosedembodiments.

FIG. 33A is a flowchart showing an example process for determining alocation of a landmark for use in navigation of an autonomous vehicleconsistent with the disclosed embodiments.

FIG. 33B is a flowchart showing an example process for measuring aposition of a landmark for use in navigation of an autonomous vehicleconsistent with the disclosed embodiments.

FIG. 34 is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich the vehicle navigates using a landmark.

FIG. 35 is another diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich the vehicle navigates using a landmark.

FIG. 36 is a flowchart showing an exemplary process for navigating anexemplary vehicle using a landmark.

FIG. 37 is a diagrammatic top view representation of an exemplaryautonomous vehicle including a system consistent with the disclosedembodiments in which the autonomous vehicle navigates using tailalignment.

FIG. 38 is another diagrammatic top view representation of an exemplaryautonomous vehicle including a system consistent with the disclosedembodiments in which the autonomous vehicle navigates using tailalignment.

FIG. 39 is a flowchart showing an exemplary process for navigating anexemplary autonomous vehicle using tail alignment.

FIG. 40 is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich the vehicle navigates road junctions using two or more landmarks.

FIG. 41 is a flowchart showing an exemplary process for navigating anexemplary vehicle over road junctions using two or more landmarks.

FIG. 42 is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich the vehicle navigates using overlapping maps.

FIGS. 43A, 43B, and 43C are flowcharts showing an exemplary process fornavigating an exemplary vehicle using overlapping maps.

FIG. 44 shows an exemplary remote server in communication a vehicle,consistent with the disclosed embodiments.

FIG. 45 shows a vehicle navigating along a multi-lane road, consistentwith disclosed embodiments.

FIG. 46 shows a vehicle navigating using target trajectories along amulti-lane road, consistent with disclosed embodiments.

FIG. 47 shows an example of a road signature profile, consistent withthe disclosed embodiments.

FIG. 48 illustrates an exemplary environment, consistent with thedisclosed embodiments.

FIG. 49 is a flow chart showing an exemplary process for sparse mapautonomous vehicle navigation, consistent with the disclosed embodiments

FIG. 50 illustrates an example environment for autonomous navigationbased on an expected landmark location consistent with the disclosedembodiments.

FIG. 51 illustrates a configuration for autonomous navigation consistentwith the disclosed embodiments.

FIG. 52 illustrates another example environment for autonomousnavigation based on an expected landmark location consistent with thedisclosed embodiments.

FIG. 53 illustrates another example environment for autonomousnavigation based on an expected landmark location consistent with thedisclosed embodiments.

FIG. 54 is a flow chart showing an exemplary process for autonomousnavigation based on an expected landmark location consistent with thedisclosed embodiments.

FIG. 55 is a diagrammatic representation of exemplary vehicle controlsystems consistent with the disclosed embodiments.

FIG. 56 is diagrammatic top view representation of an exemplary vehicleincluding a system consistent with the disclosed embodiments in whichthe vehicle navigates using lane width profiles or road width profiles.

FIG. 57 is graph showing an exemplary profile that may be used by thevehicle control systems consistent with the disclosed embodiments.

FIG. 58 is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich the vehicle navigates using lengths or spacings of road markingson a road segment.

FIG. 59 is a diagrammatic top view representation of an exemplaryvehicle including a system consistent with the disclosed embodiments inwhich the vehicle navigates using information regarding curvature of aroad segment.

FIG. 60 is a flowchart showing an exemplary process for navigating anexemplary vehicle using road signatures.

FIG. 61A is a diagrammatic side view representation of an exemplaryvehicle consistent with disclosed embodiments.

FIG. 61B is a diagrammatic side view representation of an exemplaryvehicle consistent with disclosed embodiments.

FIG. 62 is a diagrammatic top view representation of an exemplaryvehicle autonomously navigating on a road consistent with disclosedembodiments.

FIG. 63 is a flowchart showing an exemplary process for autonomouslynavigating a vehicle consistent with disclosed embodiments.

FIG. 64 is a diagrammatic perspective view of an environment captured bya forward facing image capture device on an exemplary vehicle consistentwith disclosed embodiments.

FIG. 65 is an exemplary image received from a forward facing imagecapture device of a vehicle consistent with disclosed embodiments.

FIG. 66 is a flowchart showing an exemplary process for navigating avehicle by determining a free space region in which the vehicle cantravel consistent with disclosed embodiments.

FIG. 67 is a diagrammatic top view representation of an exemplaryvehicle navigating on a road with snow covering at least some lanemarkings and road edges consistent with disclosed embodiments.

FIG. 68 is a flowchart showing an exemplary process for navigating avehicle on a road with snow covering at least some lane markings androad edges consistent with disclosed embodiments.

FIG. 69 is a diagrammatic top view representation of an exemplaryvehicle including a system for calibrating a speed of the vehicleconsistent with disclosed embodiments.

FIG. 70 is a flowchart showing an exemplary process for calibrating aspeed of a vehicle consistent with disclosed embodiments.

FIG. 71 is another diagrammatic top view representation of an exemplaryvehicle including a system for calibrating a speed of the vehicleconsistent with disclosed embodiments.

FIG. 72 is a flowchart showing another exemplary process for calibratinga speed of a vehicle consistent with disclosed embodiments.

FIG. 73 is an illustration of a street view of an exemplary roadsegment, consistent with disclosed embodiments.

FIG. 74 is an illustration of birds-eye view of an exemplary roadsegment, consistent with disclosed embodiments.

FIG. 75 is a flowchart showing an exemplary process for determining alane assignment for a vehicle, consistent with disclosed embodiments.

FIG. 76 is an illustration of a street view of an exemplary roadsegment, consistent with disclosed embodiments.

FIG. 77A is an illustration of birds-eye view of an exemplary roadsegment, consistent with disclosed embodiments.

FIG. 77B is an illustration of a street view of an exemplary roadsegment consistent with disclosed embodiments.

FIG. 78 is a flowchart showing an exemplary process for autonomouslynavigating a vehicle along a road segment, consistent with disclosedembodiments.

FIG. 79A illustrates a plan view of a vehicle traveling on a roadwayapproaching wintery and icy road conditions at a particular locationconsistent with disclosed embodiments.

FIG. 79B illustrates a plan view of a vehicle traveling on a roadwayapproaching a pedestrian consistent with disclosed embodiments.

FIG. 79C illustrates a plan view of a vehicle traveling on a roadway inclose proximity to another vehicle consistent with disclosedembodiments.

FIG. 79D illustrates a plan view of a vehicle traveling on a roadway ina lane that is ending consistent with disclosed embodiments.

FIG. 80 illustrates a diagrammatic side view representation of anexemplary vehicle including the system consistent with the disclosedembodiments.

FIG. 81 illustrates an example flowchart representing a method foradaptive navigation of a vehicle based on user intervention consistentwith the disclosed embodiments.

FIG. 82A illustrates a plan view of a vehicle traveling on a roadwaywith a parked car consistent with disclosed embodiments.

FIG. 82B illustrates a plan view of a vehicle traveling on a roadway ina lane that is ending consistent with the disclosed embodiments.

FIG. 82C illustrates a plan view of a vehicle traveling on a roadwayapproaching a pedestrian consistent with disclosed embodiments.

FIG. 82D illustrates a plan view of a vehicle traveling on a roadwayapproaching an area of construction consistent with the disclosedembodiments.

FIG. 83 illustrates an example flowchart representing a method forself-aware navigation of a vehicle consistent with the disclosedembodiments.

FIG. 84A illustrates a plan view of a vehicle traveling on a roadwaywith multiple parked cars consistent with the disclosed embodiments.

FIG. 84B illustrates a plan view of a vehicle traveling on a roadwaywith a car intervening directly in front of the vehicle consistent withthe disclosed embodiments.

FIG. 84C illustrates a plan view of a vehicle traveling on a roadwaywith a temporary barrier directly in front of the vehicle consistentwith the disclosed embodiments.

FIG. 84D illustrates a plan view of a vehicle traveling on a roadwaywith temporary roadwork directly in front of the vehicle consistent withthe disclosed embodiments.

FIG. 85A illustrates a plan view of a vehicle traveling on a roadwaywith a pot hole directly in front of the vehicle consistent with thedisclosed embodiments.

FIG. 85B illustrates a plan view of a vehicle traveling on a roadwaywith an animal and a pedestrian crossing in front of a vehicleconsistent with the disclosed embodiments.

FIG. 86 illustrates an example flowchart representing a method for anadaptive road model manager consistent with disclosed embodiments.

FIG. 87A illustrates a plan view of a single vehicle traveling on aninterstate roadway consistent with the disclosed embodiments.

FIG. 87B illustrates a plan view of a group of vehicles traveling on acity roadway consistent with the disclosed embodiments.

FIG. 87C illustrates a plan view of a vehicle traveling on a roadwaywithin a particular rural geographic region consistent with thedisclosed embodiments.

FIG. 87D illustrates a plan view of a vehicle traveling on a roadwaywith a lane shift consistent with the disclosed embodiments.

FIG. 88 illustrates an example flowchart representing a method for roadmodel management based on selective feedback consistent with thedisclosed embodiments.

DETAILED DESCRIPTION

The following detailed description refers to the accompanying drawings.Wherever possible, the same reference numbers are used in the drawingsand the following description to refer to the same or similar parts.While several illustrative embodiments are described herein,modifications, adaptations and other implementations are possible. Forexample, substitutions, additions or modifications may be made to thecomponents illustrated in the drawings, and the illustrative methodsdescribed herein may be modified by substituting, reordering, removing,or adding steps to the disclosed methods. Accordingly, the followingdetailed description is not limited to the disclosed embodiments andexamples. Instead, the proper scope is defined by the appended claims.

Autonomous Vehicle Overview

As used throughout this disclosure, the term “autonomous vehicle” refersto a vehicle capable of implementing at least one navigational changewithout driver input. A “navigational change” refers to a change in oneor more of steering, braking, or acceleration of the vehicle. To beautonomous, a vehicle need not be fully automatic (e.g., fully operationwithout a driver or without driver input). Rather, an autonomous vehicleincludes those that can operate under driver control during certain timeperiods and without driver control during other time periods. Autonomousvehicles may also include vehicles that control only some aspects ofvehicle navigation, such as steering (e.g., to maintain a vehicle coursebetween vehicle lane constraints), but may leave other aspects to thedriver (e.g., braking). In some cases, autonomous vehicles may handlesome or all aspects of braking, speed control, and/or steering of thevehicle.

As human drivers typically rely on visual cues and observations order tocontrol a vehicle, transportation infrastructures are built accordingly,with lane markings, traffic signs, and traffic lights are all designedto provide visual information to drivers. In view of these designcharacteristics of transportation infrastructures, an autonomous vehiclemay include a camera and a processing unit that analyzes visualinformation captured from the environment of the vehicle. The visualinformation may include, for example, components of the transportationinfrastructure (e.g., lane markings, traffic signs, traffic lights,etc.) that are observable by drivers and other obstacles (e.g., othervehicles, pedestrians, debris, etc.). Additionally, an autonomousvehicle may also use stored information, such as information thatprovides a model of the vehicle's environment when navigating. Forexample, the vehicle may use GPS data, sensor data (e.g., from anaccelerometer, a speed sensor, a suspension sensor, etc.), and/or othermap data to provide information related to its environment while it istraveling, and the vehicle (as well as other vehicles) may use theinformation to localize itself on the model.

In some embodiments in this disclosure, an autonomous vehicle may useinformation obtained while navigating (e.g., from a camera, GPS device,an accelerometer, a speed sensor, a suspension sensor, etc.). In otherembodiments, an autonomous vehicle may use information obtained frompast navigations by the vehicle (or by other vehicles) while navigating.In yet other embodiments, an autonomous vehicle may use a combination ofinformation obtained while navigating and information obtained from pastnavigations. The following sections provide an overview of a systemconsistent with the disclosed embodiments, following by an overview of aforward-facing imaging system and methods consistent with the system.The sections that follow disclose systems and methods for constructing,using, and updating a sparse map for autonomous vehicle navigation.

System Overview

FIG. 1 is a block diagram representation of a system 100 consistent withthe exemplary disclosed embodiments. System 100 may include variouscomponents depending on the requirements of a particular implementation.In some embodiments, system 100 may include a processing unit 110, animage acquisition unit 120, a position sensor 130, one or more memoryunits 140, 150, a map database 160, a user interface 170, and a wirelesstransceiver 172. Processing unit 110 may include one or more processingdevices. In some embodiments, processing unit 110 may include anapplications processor 180, an image processor 190, or any othersuitable processing device. Similarly, image acquisition unit 120 mayinclude any number of image acquisition devices and components dependingon the requirements of a particular application. In some embodiments,image acquisition unit 120 may include one or more image capture devices(e.g., cameras), such as image capture device 122, image capture device124, and image capture device 126. System 100 may also include a datainterface 128 communicatively connecting processing device 110 to imageacquisition device 120. For example, data interface 128 may include anywired and/or wireless link or links for transmitting image data acquiredby image accusation device 120 to processing unit 110.

Wireless transceiver 172 may include one or more devices configured toexchange transmissions over an air interface to one or more networks(e.g., cellular, the Internet, etc.) by use of a radio frequency,infrared frequency, magnetic field, or an electric field. Wirelesstransceiver 172 may use any known standard to transmit and/or receivedata (e.g., Wi-Fi, Bluetooth®, Bluetooth Smart, 802.15.4, ZigBee, etc.).

Both applications processor 180 and image processor 190 may includevarious types of processing devices. For example, either or both ofapplications processor 180 and image processor 190 may include amicroprocessor, preprocessors (such as an image preprocessor), graphicsprocessors, a central processing unit (CPU), support circuits, digitalsignal processors, integrated circuits, memory, or any other types ofdevices suitable for running applications and for image processing andanalysis. In some embodiments, applications processor 180 and/or imageprocessor 190 may include any type of single or multi-core processor,mobile device microcontroller, central processing unit, etc. Variousprocessing devices may be used, including, for example, processorsavailable from manufacturers such as Intel®, AMD®, etc. and may includevarious architectures (e.g., x86 processor, ARM®, etc.).

In some embodiments, applications processor 180 and/or image processor190 may include any of the EyeQ series of processor chips available fromMobileye®. These processor designs each include multiple processingunits with local memory and instruction sets. Such processors mayinclude video inputs for receiving image data from multiple imagesensors and may also include video out capabilities. In one example, theEyeQ2® uses 90 nm-micron technology operating at 332 Mhz. The EyeQ2®architecture consists of two floating point, hyper-thread 32-bit RISCCPUs (MIPS32® 34K® cores), five Vision Computing Engines (VCE), threeVector Microcode Processors (VMP®), Denali 64-bit Mobile DDR Controller,128-bit internal Sonics Interconnect, dual 16-bit Video input and 18-bitVideo output controllers, 16 channels DMA and several peripherals. TheMIPS34K CPU manages the five VCEs, three VMP™ and the DMA, the secondMIPS34K CPU and the multi-channel DMA as well as the other peripherals.The five VCEs, three VMP® and the MIPS34K CPU can perform intensivevision computations required by multi-function bundle applications. Inanother example, the EycQ3®, which is a third generation processor andis six times more powerful that the EyeQ2®, may be used in the disclosedembodiments.

Any of the processing devices disclosed herein may be configured toperform certain functions. Configuring a processing device, such as anyof the described EyeQ processors or other controller or microprocessor,to perform certain functions may include programming of computerexecutable instructions and making those instructions available to theprocessing device for execution during operation of the processingdevice. In some embodiments, configuring a processing device may includeprogramming the processing device directly with architecturalinstructions. In other embodiments, configuring a processing device mayinclude storing executable instructions on a memory that is accessibleto the processing device during operation. For example, the processingdevice may access the memory to obtain and execute the storedinstructions during operation.

While FIG. 1 depicts two separate processing devices included inprocessing unit 110, more or fewer processing devices may be used. Forexample, in some embodiments, a single processing device may be used toaccomplish the tasks of applications processor 180 and image processor190. In other embodiments, these tasks may be performed by more than twoprocessing devices. Further, in some embodiments, system 100 may includeone or more of processing unit 110 without including other components,such as image acquisition unit 120.

Processing unit 110 may comprise various types of devices. For example,processing unit 110 may include various devices, such as a controller,an image preprocessor, a central processing unit (CPU), supportcircuits, digital signal processors, integrated circuits, memory, or anyother types of devices for image processing and analysis. The imagepreprocessor may include a video processor for capturing, digitizing andprocessing the imagery from the image sensors. The CPU may comprise anynumber of microcontrollers or microprocessors. The support circuits maybe any number of circuits generally well known in the art, includingcache, power supply, clock and input-output circuits. The memory maystore software that, when executed by the processor, controls theoperation of the system. The memory may include databases and imageprocessing software. The memory may comprise any number of random accessmemories, read only memories, flash memories, disk drives, opticalstorage, tape storage, removable storage and other types of storage. Inone instance, the memory may be separate from the processing unit 110.In another instance, the memory may be integrated into the processingunit 110.

Each memory 140, 150 may include software instructions that whenexecuted by a processor (e.g., applications processor 180 and/or imageprocessor 190), may control operation of various aspects of system 100.These memory units may include various databases and image processingsoftware. The memory units may include random access memory, read onlymemory, flash memory, disk drives, optical storage, tape storage,removable storage and/or any other types of storage. In someembodiments, memory units 140, 150 may be separate from the applicationsprocessor 180 and/or image processor 190. In other embodiments, thesememory units may be integrated into applications processor 180 and/orimage processor 190.

Position sensor 130 may include any type of device suitable fordetermining a location associated with at least one component of system100. In some embodiments, position sensor 130 may include a GPSreceiver. Such receivers can determine a user position and velocity byprocessing signals broadcasted by global positioning system satellites.Position information from position sensor 130 may be made available toapplications processor 180 and/or image processor 190.

In some embodiments, system 100 may include components such as a speedsensor (e.g., a tachometer) for measuring a speed of vehicle 200 and/oran accelerometer for measuring acceleration of vehicle 200.

User interface 170 may include any device suitable for providinginformation to or for receiving inputs from one or more users of system100. In some embodiments, user interface 170 may include user inputdevices, including, for example, a touchscreen, microphone, keyboard,pointer devices, track wheels, cameras, knobs, buttons, etc. With suchinput devices, a user may be able to provide information inputs orcommands to system 100 by typing instructions or information, providingvoice commands, selecting menu options on a screen using buttons,pointers, or eye-tracking capabilities, or through any other suitabletechniques for communicating information to system 100.

User interface 170 may be equipped with one or more processing devicesconfigured to provide and receive information to or from a user andprocess that information for use by, for example, applications processor180. In some embodiments, such processing devices may executeinstructions for recognizing and tracking eye movements, receiving andinterpreting voice commands, recognizing and interpreting touches and/orgestures made on a touchscreen, responding to keyboard entries or menuselections, etc. In some embodiments, user interface 170 may include adisplay, speaker, tactile device, and/or any other devices for providingoutput information to a user.

Map database 160 may include any type of database for storing map datauseful to system 100. In some embodiments, map database 160 may includedata relating to the position, in a reference coordinate system, ofvarious items, including roads, water features, geographic features,businesses, points of interest, restaurants, gas stations, etc. Mapdatabase 160 may store not only the locations of such items, but alsodescriptors relating to those items, including, for example, namesassociated with any of the stored features. In some embodiments, mapdatabase 160 may be physically located with other components of system100. Alternatively or additionally, map database 160 or a portionthereof may be located remotely with respect to other components ofsystem 100 (e.g., processing unit 110). In such embodiments, informationfrom map database 160 may be downloaded over a wired or wireless dataconnection to a network (e.g., over a cellular network and/or theInternet, etc.).

Image capture devices 122, 124, and 126 may each include any type ofdevice suitable for capturing at least one image from an environment.Moreover, any number of image capture devices may be used to acquireimages for input to the image processor. Some embodiments may includeonly a single image capture device, while other embodiments may includetwo, three, or even four or more image capture devices. Image capturedevices 122, 124, and 126 will be further described with reference toFIGS. 2B-2E, below.

System 100, or various components thereof, may be incorporated intovarious different platforms. In some embodiments, system 100 may beincluded on a vehicle 200, as shown in FIG. 2A. For example, vehicle 200may be equipped with a processing unit 110 and any of the othercomponents of system 100, as described above relative to FIG. 1. Whilein some embodiments vehicle 200 may be equipped with only a single imagecapture device (e.g., camera), in other embodiments, such as thosediscussed in connection with FIGS. 2B-2E, multiple image capture devicesmay be used. For example, either of image capture devices 122 and 124 ofvehicle 200, as shown in FIG. 2A, may be part of an ADAS (AdvancedDriver Assistance Systems) imaging set.

The image capture devices included on vehicle 200 as part of the imageacquisition unit 120 may be positioned at any suitable location. In someembodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122may be located in the vicinity of the rearview mirror. This position mayprovide a line of sight similar to that of the driver of vehicle 200,which may aid in determining what is and is not visible to the driver.Image capture device 122 may be positioned at any location near therearview mirror, but placing image capture device 122 on the driver sideof the mirror may further aid in obtaining images representative of thedriver's field of view and/or line of sight.

Other locations for the image capture devices of image acquisition unit120 may also be used. For example, image capture device 124 may belocated on or in a bumper of vehicle 200. Such a location may beespecially suitable for image capture devices having a wide field ofview. The line of sight of bumper-located image capture devices can bedifferent from that of the driver and, therefore, the bumper imagecapture device and driver may not always see the same objects. The imagecapture devices (e.g., image capture devices 122, 124, and 126) may alsobe located in other locations. For example, the image capture devicesmay be located on or in one or both of the side mirrors of vehicle 200,on the roof of vehicle 200, on the hood of vehicle 200, on the trunk ofvehicle 200, on the sides of vehicle 200, mounted on, positioned behind,or positioned in front of any of the windows of vehicle 200, and mountedin or near light figures on the front and/or back of vehicle 200, etc.

In addition to image capture devices, vehicle 200 may include variousother components of system 100. For example, processing unit 110 may beincluded on vehicle 200 either integrated with or separate from anengine control unit (ECU) of the vehicle. Vehicle 200 may also beequipped with a position sensor 130, such as a GPS receiver and may alsoinclude a map database 160 and memory units 140 and 150.

As discussed earlier, wireless transceiver 172 may and/or receive dataover one or more networks (e.g., cellular networks, the Internet, etc.).For example, wireless transceiver 172 may upload data collected bysystem 100 to one or more servers, and download data from the one ormore servers. Via wireless transceiver 172, system 100 may receive, forexample, periodic or on demand updates to data stored in map database160, memory 140, and/or memory 150. Similarly, wireless transceiver 172may upload any data (e.g., images captured by image acquisition unit120, data received by position sensor 130 or other sensors, vehiclecontrol systems, etc.) from by system 100 and/or any data processed byprocessing unit 110 to the one or more servers.

System 100 may upload data to a server (e.g., to the cloud) based on aprivacy level setting. For example, system 100 may implement privacylevel settings to regulate or limit the types of data (includingmetadata) sent to the server that may uniquely identify a vehicle and ordriver/owner of a vehicle. Such settings may be set by user via, forexample, wireless transceiver 172, be initialized by factory defaultsettings, or by data received by wireless transceiver 172.

In some embodiments, system 100 may upload data according to a “high”privacy level, and under setting a setting, system 100 may transmit data(e.g., location information related to a route, captured images, etc.)without any details about the specific vehicle and/or driver/owner. Forexample, when uploading data according to a “high” privacy setting,system 100 may not include a vehicle identification number (VIN) or aname of a driver or owner of the vehicle, and may instead of transmitdata, such as captured images and/or limited location informationrelated to a route.

Other privacy levels are contemplated. For example, system 100 maytransmit data to a server according to an “intermediate” privacy leveland include additional information not included under a “high” privacylevel, such as a make and/or model of a vehicle and/or a vehicle type(e.g., a passenger vehicle, sport utility vehicle, truck, etc.). In someembodiments, system 100 may upload data according to a “low” privacylevel. Under a “low” privacy level setting, system 100 may upload dataand include information sufficient to uniquely identify a specificvehicle, owner/driver, and/or a portion or entirely of a route traveledby the vehicle. Such “low” privacy level data may include one or moreof, for example, a VIN, a driver/owner name, an origination point of avehicle prior to departure, an intended destination of the vehicle, amake and/or model of the vehicle, a type of the vehicle, etc.

FIG. 2A is a diagrammatic side view representation of an exemplaryvehicle imaging system consistent with the disclosed embodiments. FIG.2B is a diagrammatic top view illustration of the embodiment shown inFIG. 2A. As illustrated in FIG. 2B, the disclosed embodiments mayinclude a vehicle 200 including in its body a system 100 with a firstimage capture device 122 positioned in the vicinity of the rearviewmirror and/or near the driver of vehicle 200, a second image capturedevice 124 positioned on or in a bumper region (e.g., one of bumperregions 210) of vehicle 200, and a processing unit 110.

As illustrated in FIG. 2C, image capture devices 122 and 124 may both bepositioned in the vicinity of the rearview mirror and/or near the driverof vehicle 200. Additionally, while two image capture devices 122 and124 are shown in FIGS. 2B and 2C, it should be understood that otherembodiments may include more than two image capture devices. Forexample, in the embodiments shown in FIGS. 2D and 2E, first, second, andthird image capture devices 122, 124, and 126, are included in thesystem 100 of vehicle 200.

As illustrated in FIG. 2D, image capture device 122 may be positioned inthe vicinity of the rearview mirror and/or near the driver of vehicle200, and image capture devices 124 and 126 may be positioned on or in abumper region (e.g., one of bumper regions 210) of vehicle 200. And asshown in FIG. 2E, image capture devices 122, 124, and 126 may bepositioned in the vicinity of the rearview mirror and/or near the driverseat of vehicle 200. The disclosed embodiments are not limited to anyparticular number and configuration of the image capture devices, andthe image capture devices may be positioned in any appropriate locationwithin and/or on vehicle 200.

It is to be understood that the disclosed embodiments are not limited tovehicles and could be applied in other contexts. It is also to beunderstood that disclosed embodiments are not limited to a particulartype of vehicle 200 and may be applicable to all types of vehiclesincluding automobiles, trucks, trailers, and other types of vehicles.

The first image capture device 122 may include any suitable type ofimage capture device. Image capture device 122 may include an opticalaxis. In one instance, the image capture device 122 may include anAptina M9V024 WVGA sensor with a global shutter. In other embodiments,image capture device 122 may provide a resolution of 1280×960 pixels andmay include a rolling shutter. Image capture device 122 may includevarious optical elements. In some embodiments one or more lenses may beincluded, for example, to provide a desired focal length and field ofview for the image capture device. In some embodiments, image capturedevice 122 may be associated with a 6 mm lens or a 12 mm lens. In someembodiments, image capture device 122 may be configured to captureimages having a desired field-of-view (FOV) 202, as illustrated in FIG.2D. For example, image capture device 122 may be configured to have aregular FOV, such as within a range of 40 degrees to 56 degrees,including a 46 degree FOV, 50 degree FOV, 52 degree FOV, or greater.Alternatively, image capture device 122 may be configured to have anarrow FOV in the range of 23 to 40 degrees, such as a 28 degree FOV or36 degree FOV. In addition, image capture device 122 may be configuredto have a wide FOV in the range of 100 to 180 degrees. In someembodiments, image capture device 122 may include a wide angle bumpercamera or one with up to a 180 degree FOV. In some embodiments, imagecapture device 122 may be a 7.2 M pixel image capture device with anaspect ratio of about 2:1 (e.g., H×V=3800×1900 pixels) with about 100degree horizontal FOV. Such an image capture device may be used in placeof a three image capture device configuration. Due to significant lensdistortion, the vertical FOV of such an image capture device may besignificantly less than 50 degrees in implementations in which the imagecapture device uses a radially symmetric lens. For example, such a lensmay not be radially symmetric which would allow for a vertical FOVgreater than 50 degrees with 100 degree horizontal FOV.

The first image capture device 122 may acquire a plurality of firstimages relative to a scene associated with the vehicle 200. Each of theplurality of first images may be acquired as a series of image scanlines, which may be captured using a rolling shutter. Each scan line mayinclude a plurality of pixels.

The first image capture device 122 may have a scan rate associated withacquisition of each of the first series of image scan lines. The scanrate may refer to a rate at which an image sensor can acquire image dataassociated with each pixel included in a particular scan line.

Image capture devices 122, 124, and 126 may contain any suitable typeand number of image sensors, including CCD sensors or CMOS sensors, forexample. In one embodiment, a CMOS image sensor may be employed alongwith a rolling shutter, such that each pixel in a row is read one at atime, and scanning of the rows proceeds on a row-by-row basis until anentire image frame has been captured. In some embodiments, the rows maybe captured sequentially from top to bottom relative to the frame.

In some embodiments, one or more of the image capture devices (e.g.,image capture devices 122, 124, and 126) disclosed herein may constitutea high resolution imager and may have a resolution greater than 5 Mpixel, 7 M pixel, 10 M pixel, or greater.

The use of a rolling shutter may result in pixels in different rowsbeing exposed and captured at different times, which may cause skew andother image artifacts in the captured image frame. On the other hand,when the image capture device 122 is configured to operate with a globalor synchronous shutter, all of the pixels may be exposed for the sameamount of time and during a common exposure period. As a result, theimage data in a frame collected from a system employing a global shutterrepresents a snapshot of the entire FOV (such as FOV 202) at aparticular time. In contrast, in a rolling shutter application, each rowin a frame is exposed and data is capture at different times. Thus,moving objects may appear distorted in an image capture device having arolling shutter. This phenomenon will be described in greater detailbelow.

The second image capture device 124 and the third image capturing device126 may be any type of image capture device. Like the first imagecapture device 122, each of image capture devices 124 and 126 mayinclude an optical axis. In one embodiment, each of image capturedevices 124 and 126 may include an Aptina M9V024 WVGA sensor with aglobal shutter. Alternatively, each of image capture devices 124 and 126may include a rolling shutter. Like image capture device 122, imagecapture devices 124 and 126 may be configured to include various lensesand optical elements. In some embodiments, lenses associated with imagecapture devices 124 and 126 may provide FOVs (such as FOVs 204 and 206)that are the same as, or narrower than, a FOV (such as FOV 202)associated with image capture device 122. For example, image capturedevices 124 and 126 may have FOVs of 40 degrees, 30 degrees, 26 degrees,23 degrees, 20 degrees, or less.

Image capture devices 124 and 126 may acquire a plurality of second andthird images relative to a scene associated with the vehicle 200. Eachof the plurality of second and third images may be acquired as a secondand third series of image scan lines, which may be captured using arolling shutter. Each scan line or row may have a plurality of pixels.Image capture devices 124 and 126 may have second and third scan ratesassociated with acquisition of each of image scan lines included in thesecond and third series.

Each image capture device 122, 124, and 126 may be positioned at anysuitable position and orientation relative to vehicle 200. The relativepositioning of the image capture devices 122, 124, and 126 may beselected to aid in fusing together the information acquired from theimage capture devices. For example, in some embodiments, a FOV (such asFOV 204) associated with image capture device 124 may overlap partiallyor fully with a FOV (such as FOV 202) associated with image capturedevice 122 and a FOV (such as FOV 206) associated with image capturedevice 126.

Image capture devices 122, 124, and 126 may be located on vehicle 200 atany suitable relative heights. In one instance, there may be a heightdifference between the image capture devices 122, 124, and 126, whichmay provide sufficient parallax information to enable stereo analysis.For example, as shown in FIG. 2A, the two image capture devices 122 and124 are at different heights. There may also be a lateral displacementdifference between image capture devices 122, 124, and 126, givingadditional parallax information for stereo analysis by processing unit110, for example. The difference in the lateral displacement may bedenoted by d_(x), as shown in FIGS. 2C and 2D. In some embodiments, foreor aft displacement (e.g., range displacement) may exist between imagecapture devices 122, 124, and 126. For example, image capture device 122may be located 0.5 to 2 meters or more behind image capture device 124and/or image capture device 126. This type of displacement may enableone of the image capture devices to cover potential blind spots of theother image capture device(s).

Image capture devices 122 may have any suitable resolution capability(e.g., number of pixels associated with the image sensor), and theresolution of the image sensor(s) associated with the image capturedevice 122 may be higher, lower, or the same as the resolution of theimage sensor(s) associated with image capture devices 124 and 126. Insome embodiments, the image sensor(s) associated with image capturedevice 122 and/or image capture devices 124 and 126 may have aresolution of 640×480, 1024×768, 1280×960, or any other suitableresolution.

The frame rate (e.g., the rate at which an image capture device acquiresa set of pixel data of one image frame before moving on to capture pixeldata associated with the next image frame) may be controllable. Theframe rate associated with image capture device 122 may be higher,lower, or the same as the frame rate associated with image capturedevices 124 and 126. The frame rate associated with image capturedevices 122, 124, and 126 may depend on a variety of factors that mayaffect the timing of the frame rate. For example, one or more of imagecapture devices 122, 124, and 126 may include a selectable pixel delayperiod imposed before or after acquisition of image data associated withone or more pixels of an image sensor in image capture device 122, 124,and/or 126. Generally, image data corresponding to each pixel may beacquired according to a clock rate for the device (e.g., one pixel perclock cycle). Additionally, in embodiments including a rolling shutter,one or more of image capture devices 122, 124, and 126 may include aselectable horizontal blanking period imposed before or afteracquisition of image data associated with a row of pixels of an imagesensor in image capture device 122, 124, and/or 126. Further, one ormore of image capture devices 122, 124, and/or 126 may include aselectable vertical blanking period imposed before or after acquisitionof image data associated with an image frame of image capture device122, 124, and 126.

These timing controls may enable synchronization of frame ratesassociated with image capture devices 122, 124, and 126, even where theline scan rates of each are different. Additionally, as will bediscussed in greater detail below, these selectable timing controls,among other factors (e.g., image sensor resolution, maximum line scanrates, etc.) may enable synchronization of image capture from an areawhere the FOV of image capture device 122 overlaps with one or more FOVsof image capture devices 124 and 126, even where the field of view ofimage capture device 122 is different from the FOVs of image capturedevices 124 and 126.

Frame rate timing in image capture device 122, 124, and 126 may dependon the resolution of the associated image sensors. For example, assumingsimilar line scan rates for both devices, if one device includes animage sensor having a resolution of 640×480 and another device includesan image sensor with a resolution of 1280×960, then more time will berequired to acquire a frame of image data from the sensor having thehigher resolution.

Another factor that may affect the timing of image data acquisition inimage capture devices 122, 124, and 126 is the maximum line scan rate.For example, acquisition of a row of image data from an image sensorincluded in image capture device 122, 124, and 126 will require someminimum amount of time. Assuming no pixel delay periods are added, thisminimum amount of time for acquisition of a row of image data will berelated to the maximum line scan rate for a particular device. Devicesthat offer higher maximum line scan rates have the potential to providehigher frame rates than devices with lower maximum line scan rates. Insome embodiments, one or more of image capture devices 124 and 126 mayhave a maximum line scan rate that is higher than a maximum line scanrate associated with image capture device 122. In some embodiments, themaximum line scan rate of image capture device 124 and/or 126 may be1.25, 1.5, 1.75, or 2 times or more than a maximum line scan rate ofimage capture device 122.

In another embodiment, image capture devices 122, 124, and 126 may havethe same maximum line scan rate, but image capture device 122 may beoperated at a scan rate less than or equal to its maximum scan rate. Thesystem may be configured such that one or more of image capture devices124 and 126 operate at a line scan rate that is equal to the line scanrate of image capture device 122. In other instances, the system may beconfigured such that the line scan rate of image capture device 124and/or image capture device 126 may be 1.25, 1.5, 1.75, or 2 times ormore than the line scan rate of image capture device 122.

In some embodiments, image capture devices 122, 124, and 126 may beasymmetric. That is, they may include cameras having different fields ofview (FOV) and focal lengths. The fields of view of image capturedevices 122, 124, and 126 may include any desired area relative to anenvironment of vehicle 200, for example. In some embodiments, one ormore of image capture devices 122, 124, and 126 may be configured toacquire image data from an environment in front of vehicle 200, behindvehicle 200, to the sides of vehicle 200, or combinations thereof.

Further, the focal length associated with each image capture device 122,124, and/or 126 may be selectable (e.g., by inclusion of appropriatelenses etc.) such that each device acquires images of objects at adesired distance range relative to vehicle 200. For example, in someembodiments image capture devices 122, 124, and 126 may acquire imagesof close-up objects within a few meters from the vehicle. Image capturedevices 122, 124, and 126 may also be configured to acquire images ofobjects at ranges more distant from the vehicle (e.g., 25 m, 50 m, 100m, 150 m, or more). Further, the focal lengths of image capture devices122, 124, and 126 may be selected such that one image capture device(e.g., image capture device 122) can acquire images of objectsrelatively close to the vehicle (e.g., within 10 m or within 20 m) whilethe other image capture devices (e.g., image capture devices 124 and126) can acquire images of more distant objects (e.g., greater than 20m, 50 m, 100 m, 150 m, etc.) from vehicle 200.

According to some embodiments, the FOV of one or more image capturedevices 122, 124, and 126 may have a wide angle. For example, it may beadvantageous to have a FOV of 140 degrees, especially for image capturedevices 122, 124, and 126 that may be used to capture images of the areain the vicinity of vehicle 200. For example, image capture device 122may be used to capture images of the area to the right or left ofvehicle 200 and, in such embodiments, it may be desirable for imagecapture device 122 to have a wide FOV (e.g., at least 140 degrees).

The field of view associated with each of image capture devices 122,124, and 126 may depend on the respective focal lengths. For example, asthe focal length increases, the corresponding field of view decreases.

Image capture devices 122, 124, and 126 may be configured to have anysuitable fields of view. In one particular example, image capture device122 may have a horizontal FOV of 46 degrees, image capture device 124may have a horizontal FOV of 23 degrees, and image capture device 126may have a horizontal FOV in between 23 and 46 degrees. In anotherinstance, image capture device 122 may have a horizontal FOV of 52degrees, image capture device 124 may have a horizontal FOV of 26degrees, and image capture device 126 may have a horizontal FOV inbetween 26 and 52 degrees. In some embodiments, a ratio of the FOV ofimage capture device 122 to the FOVs of image capture device 124 and/orimage capture device 126 may vary from 1.5 to 2.0. In other embodiments,this ratio may vary between 1.25 and 2.25.

System 100 may be configured so that a field of view of image capturedevice 122 overlaps, at least partially or fully, with a field of viewof image capture device 124 and/or image capture device 126. In someembodiments, system 100 may be configured such that the fields of viewof image capture devices 124 and 126, for example, fall within (e.g.,are narrower than) and share a common center with the field of view ofimage capture device 122. In other embodiments, the image capturedevices 122, 124, and 126 may capture adjacent FOVs or may have partialoverlap in their FOVs. In some embodiments, the fields of view of imagecapture devices 122, 124, and 126 may be aligned such that a center ofthe narrower FOV image capture devices 124 and/or 126 may be located ina lower half of the field of view of the wider FOV device 122.

FIG. 2F is a diagrammatic representation of exemplary vehicle controlsystems, consistent with the disclosed embodiments. As indicated in FIG.2F, vehicle 200 may include throttling system 220, braking system 230,and steering system 240. System 100 may provide inputs (e.g., controlsignals) to one or more of throttling system 220, braking system 230,and steering system 240 over one or more data links (e.g., any wiredand/or wireless link or links for transmitting data). For example, basedon analysis of images acquired by image capture devices 122, 124, and/or126, system 100 may provide control signals to one or more of throttlingsystem 220, braking system 230, and steering system 240 to navigatevehicle 200 (e.g., by causing an acceleration, a turn, a lane shift,etc.). Further, system 100 may receive inputs from one or more ofthrottling system 220, braking system 230, and steering system 24indicating operating conditions of vehicle 200 (e.g., speed, whethervehicle 200 is braking and/or turning, etc.). Further details areprovided in connection with FIGS. 4-7, below.

As shown in FIG. 3A, vehicle 200 may also include a user interface 170for interacting with a driver or a passenger of vehicle 200. Forexample, user interface 170 in a vehicle application may include a touchscreen 320, knobs 330, buttons 340, and a microphone 350. A driver orpassenger of vehicle 200 may also use handles (e.g., located on or nearthe steering column of vehicle 200 including, for example, turn signalhandles), buttons (e.g., located on the steering wheel of vehicle 200),and the like, to interact with system 100. In some embodiments,microphone 350 may be positioned adjacent to a rearview mirror 310.Similarly, in some embodiments, image capture device 122 may be locatednear rearview mirror 310. In some embodiments, user interface 170 mayalso include one or more speakers 360 (e.g., speakers of a vehicle audiosystem). For example, system 100 may provide various notifications(e.g., alerts) via speakers 360.

FIGS. 3B-3D are illustrations of an exemplary camera mount 370configured to be positioned behind a rearview mirror (e.g., rearviewmirror 310) and against a vehicle windshield, consistent with disclosedembodiments. As shown in FIG. 3B, camera mount 370 may include imagecapture devices 122, 124, and 126. Image capture devices 124 and 126 maybe positioned behind a glare shield 380, which may be flush against thevehicle windshield and include a composition of film and/oranti-reflective materials. For example, glare shield 380 may bepositioned such that it aligns against a vehicle windshield having amatching slope. In some embodiments, each of image capture devices 122,124, and 126 may be positioned behind glare shield 380, as depicted, forexample, in FIG. 3D. The disclosed embodiments are not limited to anyparticular configuration of image capture devices 122, 124, and 126,camera mount 370, and glare shield 380. FIG. 3C is an illustration ofcamera mount 370 shown in FIG. 3B from a front perspective.

As will be appreciated by a person skilled in the art having the benefitof this disclosure, numerous variations and/or modifications may be madeto the foregoing disclosed embodiments. For example, not all componentsare essential for the operation of system 100. Further, any componentmay be located in any appropriate part of system 100 and the componentsmay be rearranged into a variety of configurations while providing thefunctionality of the disclosed embodiments. Therefore, the foregoingconfigurations are examples and, regardless of the configurationsdiscussed above, system 100 can provide a wide range of functionality toanalyze the surroundings of vehicle 200 and navigate vehicle 200 inresponse to the analysis.

As discussed below in further detail and consistent with variousdisclosed embodiments, system 100 may provide a variety of featuresrelated to autonomous driving and/or driver assist technology. Forexample, system 100 may analyze image data, position data (e.g., GPSlocation information), map data, speed data, and/or data from sensorsincluded in vehicle 200. System 100 may collect the data for analysisfrom, for example, image acquisition unit 120, position sensor 130, andother sensors. Further, system 100 may analyze the collected data todetermine whether or not vehicle 200 should take a certain action, andthen automatically take the determined action without humanintervention. For example, when vehicle 200 navigates without humanintervention, system 100 may automatically control the braking,acceleration, and/or steering of vehicle 200 (e.g., by sending controlsignals to one or more of throttling system 220, braking system 230, andsteering system 240). Further, system 100 may analyze the collected dataand issue warnings and/or alerts to vehicle occupants based on theanalysis of the collected data. Additional details regarding the variousembodiments that are provided by system 100 are provided below.

Forward-Facing Multi-Imaging System

As discussed above, system 100 may provide drive assist functionalitythat uses a multi-camera system. The multi-camera system may use one ormore cameras facing in the forward direction of a vehicle. In otherembodiments, the multi-camera system may include one or more camerasfacing to the side of a vehicle or to the rear of the vehicle. In oneembodiment, for example, system 100 may use a two-camera imaging system,where a first camera and a second camera (e.g., image capture devices122 and 124) may be positioned at the front and/or the sides of avehicle (e.g., vehicle 200). The first camera may have a field of viewthat is greater than, less than, or partially overlapping with, thefield of view of the second camera. In addition, the first camera may beconnected to a first image processor to perform monocular image analysisof images provided by the first camera, and the second camera may beconnected to a second image processor to perform monocular imageanalysis of images provided by the second camera. The outputs (e.g.,processed information) of the first and second image processors may becombined. In some embodiments, the second image processor may receiveimages from both the first camera and second camera to perform stereoanalysis. In another embodiment, system 100 may use a three-cameraimaging system where each of the cameras has a different field of view.Such a system may, therefore, make decisions based on informationderived from objects located at varying distances both forward and tothe sides of the vehicle. References to monocular image analysis mayrefer to instances where image analysis is performed based on imagescaptured from a single point of view (e.g., from a single camera).Stereo image analysis may refer to instances where image analysis isperformed based on two or more images captured with one or morevariations of an image capture parameter. For example, captured imagessuitable for performing stereo image analysis may include imagescaptured: from two or more different positions, from different fields ofview, using different focal lengths, along with parallax information,etc.

For example, in one embodiment, system 100 may implement a three cameraconfiguration using image capture devices 122-126. In such aconfiguration, image capture device 122 may provide a narrow field ofview (e.g., 34 degrees, or other values selected from a range of about20 to 45 degrees, etc.), image capture device 124 may provide a widefield of view (e.g., 150 degrees or other values selected from a rangeof about 100 to about 180 degrees), and image capture device 126 mayprovide an intermediate field of view (e.g., 46 degrees or other valuesselected from a range of about 35 to about 60 degrees). In someembodiments, image capture device 126 may act as a main or primarycamera. Image capture devices 122-126 may be positioned behind rearviewmirror 310 and positioned substantially side-by-side (e.g., 6 cm apart).Further, in some embodiments, as discussed above, one or more of imagecapture devices 122-126 may be mounted behind glare shield 380 that isflush with the windshield of vehicle 200. Such shielding may act tominimize the impact of any reflections from inside the car on imagecapture devices 122-126.

In another embodiment, as discussed above in connection with FIGS. 3Band 3C, the wide field of view camera (e.g., image capture device 124 inthe above example) may be mounted lower than the narrow and main fieldof view cameras (e.g., image devices 122 and 126 in the above example).This configuration may provide a free line of sight from the wide fieldof view camera. To reduce reflections, the cameras may be mounted closeto the windshield of vehicle 200, and may include polarizers on thecameras to damp reflected light.

A three camera system may provide certain performance characteristics.For example, some embodiments may include an ability to validate thedetection of objects by one camera based on detection results fromanother camera. In the three camera configuration discussed above,processing unit 110 may include, for example, three processing devices(e.g., three EyeQ series of processor chips, as discussed above), witheach processing device dedicated to processing images captured by one ormore of image capture devices 122-126.

In a three camera system, a first processing device may receive imagesfrom both the main camera and the narrow field of view camera, andperform vision processing of the narrow FOV camera to, for example,detect other vehicles, pedestrians, lane marks, traffic signs, trafficlights, and other road objects. Further, the first processing device maycalculate a disparity of pixels between the images from the main cameraand the narrow camera and create a 3D reconstruction of the environmentof vehicle 200. The first processing device may then combine the 3Dreconstruction with 3D map data or with 3D information calculated basedon information from another camera.

The second processing device may receive images from main camera andperform vision processing to detect other vehicles, pedestrians, lanemarks, traffic signs, traffic lights, and other road objects.Additionally, the second processing device may calculate a cameradisplacement and, based on the displacement, calculate a disparity ofpixels between successive images and create a 3D reconstruction of thescene (e.g., a structure from motion). The second processing device maysend the structure from motion based 3D reconstruction to the firstprocessing device to be combined with the stereo 3D images.

The third processing device may receive images from the wide FOV cameraand process the images to detect vehicles, pedestrians, lane marks,traffic signs, traffic lights, and other road objects. The thirdprocessing device may further execute additional processing instructionsto analyze images to identify objects moving in the image, such asvehicles changing lanes, pedestrians, etc.

In some embodiments, having streams of image-based information capturedand processed independently may provide an opportunity for providingredundancy in the system. Such redundancy may include, for example,using a first image capture device and the images processed from thatdevice to validate and/or supplement information obtained by capturingand processing image information from at least a second image capturedevice.

In some embodiments, system 100 may use two image capture devices (e.g.,image capture devices 122 and 124) in providing navigation assistancefor vehicle 200 and use a third image capture device (e.g., imagecapture device 126) to provide redundancy and validate the analysis ofdata received from the other two image capture devices. For example, insuch a configuration, image capture devices 122 and 124 may provideimages for stereo analysis by system 100 for navigating vehicle 200,while image capture device 126 may provide images for monocular analysisby system 100 to provide redundancy and validation of informationobtained based on images captured from image capture device 122 and/orimage capture device 124. That is, image capture device 126 (and acorresponding processing device) may be considered to provide aredundant sub-system for providing a check on the analysis derived fromimage capture devices 122 and 124 (e.g., to provide an automaticemergency braking (AEB) system).

One of skill in the art will recognize that the above cameraconfigurations, camera placements, number of cameras, camera locations,etc., are examples only. These components and others described relativeto the overall system may be assembled and used in a variety ofdifferent configurations without departing from the scope of thedisclosed embodiments. Further details regarding usage of a multi-camerasystem to provide driver assist and/or autonomous vehicle functionalityfollow below.

FIG. 4 is an exemplary functional block diagram of memory 140 and/or150, which may be stored/programmed with instructions for performing oneor more operations consistent with the disclosed embodiments. Althoughthe following refers to memory 140, one of skill in the art willrecognize that instructions may be stored in memory 140 and/or 150.

As shown in FIG. 4, memory 140 may store a monocular image analysismodule 402, a stereo image analysis module 404, a velocity andacceleration module 406, and a navigational response module 408. Thedisclosed embodiments are not limited to any particular configuration ofmemory 140. Further, application processor 180 and/or image processor190 may execute the instructions stored in any of modules 402-408included in memory 140. One of skill in the art will understand thatreferences in the following discussions to processing unit 110 may referto application processor 180 and image processor 190 individually orcollectively. Accordingly, steps of any of the following processes maybe performed by one or more processing devices.

In one embodiment, monocular image analysis module 402 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs monocular image analysis of a set ofimages acquired by one of image capture devices 122, 124, and 126. Insome embodiments, processing unit 110 may combine information from a setof images with additional sensory information (e.g., information fromradar) to perform the monocular image analysis. As described inconnection with FIGS. 5A-5D below, monocular image analysis module 402may include instructions for detecting a set of features within the setof images, such as lane markings, vehicles, pedestrians, road signs,highway exit ramps, traffic lights, hazardous objects, and any otherfeature associated with an environment of a vehicle. Based on theanalysis, system 100 (e.g., via processing unit 110) may cause one ormore navigational responses in vehicle 200, such as a turn, a laneshift, a change in acceleration, and the like, as discussed below inconnection with navigational response module 408.

In one embodiment, stereo image analysis module 404 may storeinstructions (such as computer vision software) which, when executed byprocessing unit 110, performs stereo image analysis of first and secondsets of images acquired by a combination of image capture devicesselected from any of image capture devices 122, 124, and 126. In someembodiments, processing unit 110 may combine information from the firstand second sets of images with additional sensory information (e.g.,information from radar) to perform the stereo image analysis. Forexample, stereo image analysis module 404 may include instructions forperforming stereo image analysis based on a first set of images acquiredby image capture device 124 and a second set of images acquired by imagecapture device 126. As described in connection with FIG. 6 below, stereoimage analysis module 404 may include instructions for detecting a setof features within the first and second sets of images, such as lanemarkings, vehicles, pedestrians, road signs, highway exit ramps, trafficlights, hazardous objects, and the like. Based on the analysis,processing unit 110 may cause one or more navigational responses invehicle 200, such as a turn, a lane shift, a change in acceleration, andthe like, as discussed below in connection with navigational responsemodule 408.

In one embodiment, velocity and acceleration module 406 may storesoftware configured to analyze data received from one or more computingand electromechanical devices in vehicle 200 that are configured tocause a change in velocity and/or acceleration of vehicle 200. Forexample, processing unit 110 may execute instructions associated withvelocity and acceleration module 406 to calculate a target speed forvehicle 200 based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include, for example, a target position, velocity, and/oracceleration, the position and/or speed of vehicle 200 relative to anearby vehicle, pedestrian, or road object, position information forvehicle 200 relative to lane markings of the road, and the like. Inaddition, processing unit 110 may calculate a target speed for vehicle200 based on sensory input (e.g., information from radar) and input fromother systems of vehicle 200, such as throttling system 220, brakingsystem 230, and/or steering system 240 of vehicle 200. Based on thecalculated target speed, processing unit 110 may transmit electronicsignals to throttling system 220, braking system 230, and/or steeringsystem 240 of vehicle 200 to trigger a change in velocity and/oracceleration by, for example, physically depressing the brake or easingup off the accelerator of vehicle 200.

In one embodiment, navigational response module 408 may store softwareexecutable by processing unit 110 to determine a desired navigationalresponse based on data derived from execution of monocular imageanalysis module 402 and/or stereo image analysis module 404. Such datamay include position and speed information associated with nearbyvehicles, pedestrians, and road objects, target position information forvehicle 200, and the like. Additionally, in some embodiments, thenavigational response may be based (partially or fully) on map data, apredetermined position of vehicle 200, and/or a relative velocity or arelative acceleration between vehicle 200 and one or more objectsdetected from execution of monocular image analysis module 402 and/orstereo image analysis module 404. Navigational response module 408 mayalso determine a desired navigational response based on sensory input(e.g., information from radar) and inputs from other systems of vehicle200, such as throttling system 220, braking system 230, and steeringsystem 240 of vehicle 200. Based on the desired navigational response,processing unit 110 may transmit electronic signals to throttling system220, braking system 230, and steering system 240 of vehicle 200 totrigger a desired navigational response by, for example, turning thesteering wheel of vehicle 200 to achieve a rotation of a predeterminedangle. In some embodiments, processing unit 110 may use the output ofnavigational response module 408 (e.g., the desired navigationalresponse) as an input to execution of velocity and acceleration module406 for calculating a change in speed of vehicle 200.

FIG. 5A is a flowchart showing an exemplary process 500A for causing oneor more navigational responses based on monocular image analysis,consistent with disclosed embodiments. At step 510, processing unit 110may receive a plurality of images via data interface 128 betweenprocessing unit 110 and image acquisition unit 120. For instance, acamera included in image acquisition unit 120 (such as image capturedevice 122 having field of view 202) may capture a plurality of imagesof an area forward of vehicle 200 (or to the sides or rear of a vehicle,for example) and transmit them over a data connection (e.g., digital,wired, USB, wireless, Bluetooth, etc.) to processing unit 110.Processing unit 110 may execute monocular image analysis module 402 toanalyze the plurality of images at step 520, as described in furtherdetail in connection with FIGS. 5B-5D below. By performing the analysis,processing unit 110 may detect a set of features within the set ofimages, such as lane markings, vehicles, pedestrians, road signs,highway exit ramps, traffic lights, and the like.

Processing unit 110 may also execute monocular image analysis module 402to detect various road hazards at step 520, such as, for example, partsof a truck tire, fallen road signs, loose cargo, small animals, and thelike. Road hazards may vary in structure, shape, size, and color, whichmay make detection of such hazards more challenging. In someembodiments, processing unit 110 may execute monocular image analysismodule 402 to perform multi-frame analysis on the plurality of images todetect road hazards. For example, processing unit 110 may estimatecamera motion between consecutive image frames and calculate thedisparities in pixels between the frames to construct a 3D-map of theroad. Processing unit 110 may then use the 3D-map to detect the roadsurface, as well as hazards existing above the road surface.

At step 530, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 520 and the techniques asdescribed above in connection with FIG. 4. Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration,and the like. In some embodiments, processing unit 110 may use dataderived from execution of velocity and acceleration module 406 to causethe one or more navigational responses. Additionally, multiplenavigational responses may occur simultaneously, in sequence, or anycombination thereof. For instance, processing unit 110 may cause vehicle200 to shift one lane over and then accelerate by, for example,sequentially transmitting control signals to steering system 240 andthrottling system 220 of vehicle 200. Alternatively, processing unit 110may cause vehicle 200 to brake while at the same time shifting lanes by,for example, simultaneously transmitting control signals to brakingsystem 230 and steering system 240 of vehicle 200.

FIG. 5B is a flowchart showing an exemplary process 500B for detectingone or more vehicles and/or pedestrians in a set of images, consistentwith disclosed embodiments. Processing unit 110 may execute monocularimage analysis module 402 to implement process 500B. At step 540,processing unit 110 may determine a set of candidate objectsrepresenting possible vehicles and/or pedestrians. For example,processing unit 110 may scan one or more images, compare the images toone or more predetermined patterns, and identify within each imagepossible locations that may contain objects of interest (e.g., vehicles,pedestrians, or portions thereof). The predetermined patterns may bedesigned in such a way to achieve a high rate of “false hits” and a lowrate of “misses.” For example, processing unit 110 may use a lowthreshold of similarity to predetermined patterns for identifyingcandidate objects as possible vehicles or pedestrians. Doing so mayallow processing unit 110 to reduce the probability of missing (e.g.,not identifying) a candidate object representing a vehicle orpedestrian.

At step 542, processing unit 110 may filter the set of candidate objectsto exclude certain candidates (e.g., irrelevant or less relevantobjects) based on classification criteria. Such criteria may be derivedfrom various properties associated with object types stored in adatabase (e.g., a database stored in memory 140). Properties may includeobject shape, dimensions, texture, position (e.g., relative to vehicle200), and the like. Thus, processing unit 110 may use one or more setsof criteria to reject false candidates from the set of candidateobjects.

At step 544, processing unit 110 may analyze multiple frames of imagesto determine whether objects in the set of candidate objects representvehicles and/or pedestrians. For example, processing unit 110 may tracka detected candidate object across consecutive frames and accumulateframe-by-frame data associated with the detected object (e.g., size,position relative to vehicle 200, etc.). Additionally, processing unit110 may estimate parameters for the detected object and compare theobject's frame-by-frame position data to a predicted position.

At step 546, processing unit 110 may construct a set of measurements forthe detected objects. Such measurements may include, for example,position, velocity, and acceleration values (relative to vehicle 200)associated with the detected objects. In some embodiments, processingunit 110 may construct the measurements based on estimation techniquesusing a series of time-based observations such as Kalman filters orlinear quadratic estimation (LQE), and/or based on available modelingdata for different object types (e.g., cars, trucks, pedestrians,bicycles, road signs, etc.). The Kalman filters may be based on ameasurement of an object's scale, where the scale measurement isproportional to a time to collision (e.g., the amount of time forvehicle 200 to reach the object). Thus, by performing steps 540-546,processing unit 110 may identify vehicles and pedestrians appearingwithin the set of captured images and derive information (e.g.,position, speed, size) associated with the vehicles and pedestrians.Based on the identification and the derived information, processing unit110 may cause one or more navigational responses in vehicle 200, asdescribed in connection with FIG. 5A, above.

At step 548, processing unit 110 may perform an optical flow analysis ofone or more images to reduce the probabilities of detecting a “falsehit” and missing a candidate object that represents a vehicle orpedestrian. The optical flow analysis may refer to, for example,analyzing motion patterns relative to vehicle 200 in the one or moreimages associated with other vehicles and pedestrians, and that aredistinct from road surface motion. Processing unit 110 may calculate themotion of candidate objects by observing the different positions of theobjects across multiple image frames, which are captured at differenttimes. Processing unit 110 may use the position and time values asinputs into mathematical models for calculating the motion of thecandidate objects. Thus, optical flow analysis may provide anothermethod of detecting vehicles and pedestrians that are nearby vehicle200. Processing unit 110 may perform optical flow analysis incombination with steps 540-546 to provide redundancy for detectingvehicles and pedestrians and increase the reliability of system 100.

FIG. 5C is a flowchart showing an exemplary process 500C for detectingroad marks and/or lane geometry information in a set of images,consistent with disclosed embodiments. Processing unit 110 may executemonocular image analysis module 402 to implement process 500C. At step550, processing unit 110 may detect a set of objects by scanning one ormore images. To detect segments of lane markings, lane geometryinformation, and other pertinent road marks, processing unit 110 mayfilter the set of objects to exclude those determined to be irrelevant(e.g., minor potholes, small rocks, etc.). At step 552, processing unit110 may group together the segments detected in step 550 belonging tothe same road mark or lane mark. Based on the grouping, processing unit110 may develop a model to represent the detected segments, such as amathematical model.

At step 554, processing unit 110 may construct a set of measurementsassociated with the detected segments. In some embodiments, processingunit 110 may create a projection of the detected segments from the imageplane onto the real-world plane. The projection may be characterizedusing a 3rd-degree polynomial having coefficients corresponding tophysical properties such as the position, slope, curvature, andcurvature derivative of the detected road. In generating the projection,processing unit 110 may take into account changes in the road surface,as well as pitch and roll rates associated with vehicle 200. Inaddition, processing unit 110 may model the road elevation by analyzingposition and motion cues present on the road surface. Further,processing unit 110 may estimate the pitch and roll rates associatedwith vehicle 200 by tracking a set of feature points in the one or moreimages.

At step 556, processing unit 110 may perform multi-frame analysis by,for example, tracking the detected segments across consecutive imageframes and accumulating frame-by-frame data associated with detectedsegments. As processing unit 110 performs multi-frame analysis, the setof measurements constructed at step 554 may become more reliable andassociated with an increasingly higher confidence level. Thus, byperforming steps 550-556, processing unit 110 may identify road marksappearing within the set of captured images and derive lane geometryinformation. Based on the identification and the derived information,processing unit 110 may cause one or more navigational responses invehicle 200, as described in connection with FIG. 5A, above.

At step 558, processing unit 110 may consider additional sources ofinformation to further develop a safety model for vehicle 200 in thecontext of its surroundings. Processing unit 110 may use the safetymodel to define a context in which system 100 may execute autonomouscontrol of vehicle 200 in a safe manner. To develop the safety model, insome embodiments, processing unit 110 may consider the position andmotion of other vehicles, the detected road edges and barriers, and/orgeneral road shape descriptions extracted from map data (such as datafrom map database 160). By considering additional sources ofinformation, processing unit 110 may provide redundancy for detectingroad marks and lane geometry and increase the reliability of system 100.

FIG. 5D is a flowchart showing an exemplary process 500D for detectingtraffic lights in a set of images, consistent with disclosedembodiments. Processing unit 110 may execute monocular image analysismodule 402 to implement process 500D. At step 560, processing unit 110may scan the set of images and identify objects appearing at locationsin the images likely to contain traffic lights. For example, processingunit 110 may filter the identified objects to construct a set ofcandidate objects, excluding those objects unlikely to correspond totraffic lights. The filtering may be done based on various propertiesassociated with traffic lights, such as shape, dimensions, texture,position (e.g., relative to vehicle 200), and the like. Such propertiesmay be based on multiple examples of traffic lights and traffic controlsignals and stored in a database. In some embodiments, processing unit110 may perform multi-frame analysis on the set of candidate objectsreflecting possible traffic lights. For example, processing unit 110 maytrack the candidate objects across consecutive image frames, estimatethe real-world position of the candidate objects, and filter out thoseobjects that are moving (which are unlikely to be traffic lights). Insome embodiments, processing unit 110 may perform color analysis on thecandidate objects and identify the relative position of the detectedcolors appearing inside possible traffic lights.

At step 562, processing unit 110 may analyze the geometry of a junction.The analysis may be based on any combination of: (i) the number of lanesdetected on either side of vehicle 200, (ii) markings (such as arrowmarks) detected on the road, and (iii) descriptions of the junctionextracted from map data (such as data from map database 160). Processingunit 110 may conduct the analysis using information derived fromexecution of monocular analysis module 402. In addition, Processing unit110 may determine a correspondence between the traffic lights detectedat step 560 and the lanes appearing near vehicle 200.

As vehicle 200 approaches the junction, at step 564, processing unit 110may update the confidence level associated with the analyzed junctiongeometry and the detected traffic lights. For instance, the number oftraffic lights estimated to appear at the junction as compared with thenumber actually appearing at the junction may impact the confidencelevel. Thus, based on the confidence level, processing unit 110 maydelegate control to the driver of vehicle 200 in order to improve safetyconditions. By performing steps 560-564, processing unit 110 mayidentify traffic lights appearing within the set of captured images andanalyze junction geometry information. Based on the identification andthe analysis, processing unit 110 may cause one or more navigationalresponses in vehicle 200, as described in connection with FIG. 5A,above.

FIG. 5E is a flowchart showing an exemplary process 500E for causing oneor more navigational responses in vehicle 200 based on a vehicle path,consistent with the disclosed embodiments. At step 570, processing unit110 may construct an initial vehicle path associated with vehicle 200.The vehicle path may be represented using a set of points expressed incoordinates (x, z), and the distance d_(i) between two points in the setof points may fall in the range of 1 to 5 meters. In one embodiment,processing unit 110 may construct the initial vehicle path using twopolynomials, such as left and right road polynomials. Processing unit110 may calculate the geometric midpoint between the two polynomials andoffset each point included in the resultant vehicle path by apredetermined offset (e.g., a smart lane offset), if any (an offset ofzero may correspond to travel in the middle of a lane). The offset maybe in a direction perpendicular to a segment between any two points inthe vehicle path. In another embodiment, processing unit 110 may use onepolynomial and an estimated lane width to offset each point of thevehicle path by half the estimated lane width plus a predeterminedoffset (e.g., a smart lane offset).

At step 572, processing unit 110 may update the vehicle path constructedat step 570. Processing unit 110 may reconstruct the vehicle pathconstructed at step 570 using a higher resolution, such that thedistance d_(k) between two points in the set of points representing thevehicle path is less than the distance d_(i) described above. Forexample, the distance d_(k) may fall in the range of 0.1 to 0.3 meters.Processing unit 110 may reconstruct the vehicle path using a parabolicspline algorithm, which may yield a cumulative distance vector Scorresponding to the total length of the vehicle path (i.e., based onthe set of points representing the vehicle path).

At step 574, processing unit 110 may determine a look-ahead point(expressed in coordinates as (x_(l), z_(l))) based on the updatedvehicle path constructed at step 572. Processing unit 110 may extractthe look-ahead point from the cumulative distance vector S, and thelook-ahead point may be associated with a look-ahead distance andlook-ahead time. The look-ahead distance, which may have a lower boundranging from 10 to 20 meters, may be calculated as the product of thespeed of vehicle 200 and the look-ahead time. For example, as the speedof vehicle 200 decreases, the look-ahead distance may also decrease(e.g., until it reaches the lower bound). The look-ahead time, which mayrange from 0.5 to 1.5 seconds, may be inversely proportional to the gainof one or more control loops associated with causing a navigationalresponse in vehicle 200, such as the heading error tracking controlloop. For example, the gain of the heading error tracking control loopmay depend on the bandwidth of a yaw rate loop, a steering actuatorloop, car lateral dynamics, and the like. Thus, the higher the gain ofthe heading error tracking control loop, the lower the look-ahead time.

At step 576, processing unit 110 may determine a heading error and yawrate command based on the look-ahead point determined at step 574.Processing unit 110 may determine the heading error by calculating thearctangent of the look-ahead point, e.g., arctan (x_(l)/z_(l)).Processing unit 110 may determine the yaw rate command as the product ofthe heading error and a high-level control gain. The high-level controlgain may be equal to: (2/look-ahead time), if the look-ahead distance isnot at the lower bound. Otherwise, the high-level control gain may beequal to: (2*speed of vehicle 200/look-ahead distance).

FIG. 5F is a flowchart showing an exemplary process 500F for determiningwhether a leading vehicle is changing lanes, consistent with thedisclosed embodiments. At step 580, processing unit 110 may determinenavigation information associated with a leading vehicle (e.g., avehicle traveling ahead of vehicle 200). For example, processing unit110 may determine the position, velocity (e.g., direction and speed),and/or acceleration of the leading vehicle, using the techniquesdescribed in connection with FIGS. 5A and 5B, above. Processing unit 110may also determine one or more road polynomials, a look-ahead point(associated with vehicle 200), and/or a snail trail (e.g., a set ofpoints describing a path taken by the leading vehicle), using thetechniques described in connection with FIG. 5E, above.

At step 582, processing unit 110 may analyze the navigation informationdetermined at step 580. In one embodiment, processing unit 110 maycalculate the distance between a snail trail and a road polynomial(e.g., along the trail). If the variance of this distance along thetrail exceeds a predetermined threshold (for example, 0.1 to 0.2 meterson a straight road, 0.3 to 0.4 meters on a moderately curvy road, and0.5 to 0.6 meters on a road with sharp curves), processing unit 110 maydetermine that the leading vehicle is likely changing lanes. In the casewhere multiple vehicles are detected traveling ahead of vehicle 200,processing unit 110 may compare the snail trails associated with eachvehicle. Based on the comparison, processing unit 110 may determine thata vehicle whose snail trail does not match with the snail trails of theother vehicles is likely changing lanes. Processing unit 110 mayadditionally compare the curvature of the snail trail (associated withthe leading vehicle) with the expected curvature of the road segment inwhich the leading vehicle is traveling. The expected curvature may beextracted from map data (e.g., data from map database 160), from roadpolynomials, from other vehicles' snail trails, from prior knowledgeabout the road, and the like. If the difference in curvature of thesnail trail and the expected curvature of the road segment exceeds apredetermined threshold, processing unit 110 may determine that theleading vehicle is likely changing lanes.

In another embodiment, processing unit 110 may compare the leadingvehicle's instantaneous position with the look-ahead point (associatedwith vehicle 200) over a specific period of time (e.g., 0.5 to 1.5seconds). If the distance between the leading vehicle's instantaneousposition and the look-ahead point varies during the specific period oftime, and the cumulative sum of variation exceeds a predeterminedthreshold (for example, 0.3 to 0.4 meters on a straight road, 0.7 to 0.8meters on a moderately curvy road, and 1.3 to 1.7 meters on a road withsharp curves), processing unit 110 may determine that the leadingvehicle is likely changing lanes. In another embodiment, processing unit110 may analyze the geometry of the snail trail by comparing the lateraldistance traveled along the trail with the expected curvature of thesnail trail. The expected radius of curvature may be determinedaccording to the calculation: (δ_(z) ²+δ_(z) ²)/2/(δ_(x)), where δ_(x)represents the lateral distance traveled and δ_(z) represents thelongitudinal distance traveled. If the difference between the lateraldistance traveled and the expected curvature exceeds a predeterminedthreshold (e.g., 500 to 700 meters), processing unit 110 may determinethat the leading vehicle is likely changing lanes. In anotherembodiment, processing unit 110 may analyze the position of the leadingvehicle. If the position of the leading vehicle obscures a roadpolynomial (e.g., the leading vehicle is overlaid on top of the roadpolynomial), then processing unit 110 may determine that the leadingvehicle is likely changing lanes. In the case where the position of theleading vehicle is such that, another vehicle is detected ahead of theleading vehicle and the snail trails of the two vehicles are notparallel, processing unit 110 may determine that the (closer) leadingvehicle is likely changing lanes.

At step 584, processing unit 110 may determine whether or not leadingvehicle 200 is changing lanes based on the analysis performed at step582. For example, processing unit 110 may make the determination basedon a weighted average of the individual analyses performed at step 582.Under such a scheme, for example, a decision by processing unit 110 thatthe leading vehicle is likely changing lanes based on a particular typeof analysis may be assigned a value of “1” (and “0” to represent adetermination that the leading vehicle is not likely changing lanes).Different analyses performed at step 582 may be assigned differentweights, and the disclosed embodiments are not limited to any particularcombination of analyses and weights.

FIG. 6 is a flowchart showing an exemplary process 600 for causing oneor more navigational responses based on stereo image analysis,consistent with disclosed embodiments. At step 610, processing unit 110may receive a first and second plurality of images via data interface128. For example, cameras included in image acquisition unit 120 (suchas image capture devices 122 and 124 having fields of view 202 and 204)may capture a first and second plurality of images of an area forward ofvehicle 200 and transmit them over a digital connection (e.g., USB,wireless, Bluetooth, etc.) to processing unit 110. In some embodiments,processing unit 110 may receive the first and second plurality of imagesvia two or more data interfaces. The disclosed embodiments are notlimited to any particular data interface configurations or protocols.

At step 620, processing unit 110 may execute stereo image analysismodule 404 to perform stereo image analysis of the first and secondplurality of images to create a 3D map of the road in front of thevehicle and detect features within the images, such as lane markings,vehicles, pedestrians, road signs, highway exit ramps, traffic lights,road hazards, and the like. Stereo image analysis may be performed in amanner similar to the steps described in connection with FIGS. 5A-5D,above. For example, processing unit 110 may execute stereo imageanalysis module 404 to detect candidate objects (e.g., vehicles,pedestrians, road marks, traffic lights, road hazards, etc.) within thefirst and second plurality of images, filter out a subset of thecandidate objects based on various criteria, and perform multi-frameanalysis, construct measurements, and determine a confidence level forthe remaining candidate objects. In performing the steps above,processing unit 110 may consider information from both the first andsecond plurality of images, rather than information from one set ofimages alone. For example, processing unit 110 may analyze thedifferences in pixel-level data (or other data subsets from among thetwo streams of captured images) for a candidate object appearing in boththe first and second plurality of images. As another example, processingunit 110 may estimate a position and/or velocity of a candidate object(e.g., relative to vehicle 200) by observing that the object appears inone of the plurality of images but not the other or relative to otherdifferences that may exist relative to objects appearing if the twoimage streams. For example, position, velocity, and/or accelerationrelative to vehicle 200 may be determined based on trajectories,positions, movement characteristics, etc. of features associated with anobject appearing in one or both of the image streams.

At step 630, processing unit 110 may execute navigational responsemodule 408 to cause one or more navigational responses in vehicle 200based on the analysis performed at step 620 and the techniques asdescribed above in connection with FIG. 4. Navigational responses mayinclude, for example, a turn, a lane shift, a change in acceleration, achange in velocity, braking, and the like. In some embodiments,processing unit 110 may use data derived from execution of velocity andacceleration module 406 to cause the one or more navigational responses.Additionally, multiple navigational responses may occur simultaneously,in sequence, or any combination thereof.

FIG. 7 is a flowchart showing an exemplary process 700 for causing oneor more navigational responses based on an analysis of three sets ofimages, consistent with disclosed embodiments. At step 710, processingunit 110 may receive a first, second, and third plurality of images viadata interface 128. For instance, cameras included in image acquisitionunit 120 (such as image capture devices 122, 124, and 126 having fieldsof view 202, 204, and 206) may capture a first, second, and thirdplurality of images of an area forward and/or to the side of vehicle 200and transmit them over a digital connection (e.g., USB, wireless,Bluetooth, etc.) to processing unit 110. In some embodiments, processingunit 110 may receive the first, second, and third plurality of imagesvia three or more data interfaces. For example, each of image capturedevices 122, 124, 126 may have an associated data interface forcommunicating data to processing unit 110. The disclosed embodiments arenot limited to any particular data interface configurations orprotocols.

At step 720, processing unit 110 may analyze the first, second, andthird plurality of images to detect features within the images, such aslane markings, vehicles, pedestrians, road signs, highway exit ramps,traffic lights, road hazards, and the like. The analysis may beperformed in a manner similar to the steps described in connection withFIGS. 5A-5D and 6, above. For instance, processing unit 110 may performmonocular image analysis (e.g., via execution of monocular imageanalysis module 402 and based on the steps described in connection withFIGS. 5A-5D, above) on each of the first, second, and third plurality ofimages. Alternatively, processing unit 110 may perform stereo imageanalysis (e.g., via execution of stereo image analysis module 404 andbased on the steps described in connection with FIG. 6, above) on thefirst and second plurality of images, the second and third plurality ofimages, and/or the first and third plurality of images. The processedinformation corresponding to the analysis of the first, second, and/orthird plurality of images may be combined. In some embodiments,processing unit 110 may perform a combination of monocular and stereoimage analyses. For example, processing unit 110 may perform monocularimage analysis (e.g., via execution of monocular image analysis module402) on the first plurality of images and stereo image analysis (e.g.,via execution of stereo image analysis module 404) on the second andthird plurality of images. The configuration of image capture devices122, 124, and 126—including their respective locations and fields ofview 202, 204, and 206—may influence the types of analyses conducted onthe first, second, and third plurality of images. The disclosedembodiments are not limited to a particular configuration of imagecapture devices 122, 124, and 126, or the types of analyses conducted onthe first, second, and third plurality of images.

In some embodiments, processing unit 110 may perform testing on system100 based on the images acquired and analyzed at steps 710 and 720. Suchtesting may provide an indicator of the overall performance of system100 for certain configurations of image capture devices 122, 124, and126. For example, processing unit 110 may determine the proportion of“false hits” (e.g., cases where system 100 incorrectly determined thepresence of a vehicle or pedestrian) and “misses.”

At step 730, processing unit 110 may cause one or more navigationalresponses in vehicle 200 based on information derived from two of thefirst, second, and third plurality of images. Selection of two of thefirst, second, and third plurality of images may depend on variousfactors, such as, for example, the number, types, and sizes of objectsdetected in each of the plurality of images. Processing unit 110 mayalso make the selection based on image quality and resolution, theeffective field of view reflected in the images, the number of capturedframes, the extent to which one or more objects of interest actuallyappear in the frames (e.g., the percentage of frames in which an objectappears, the proportion of the object that appears in each such frame,etc.), and the like.

In some embodiments, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images bydetermining the extent to which information derived from one imagesource is consistent with information derived from other image sources.For example, processing unit 110 may combine the processed informationderived from each of image capture devices 122, 124, and 126 (whether bymonocular analysis, stereo analysis, or any combination of the two) anddetermine visual indicators (e.g., lane markings, a detected vehicle andits location and/or path, a detected traffic light, etc.) that areconsistent across the images captured from each of image capture devices122, 124, and 126. Processing unit 110 may also exclude information thatis inconsistent across the captured images (e.g., a vehicle changinglanes, a lane model indicating a vehicle that is too close to vehicle200, etc.). Thus, processing unit 110 may select information derivedfrom two of the first, second, and third plurality of images based onthe determinations of consistent and inconsistent information.

Navigational responses may include, for example, a turn, a lane shift, achange in acceleration, and the like. Processing unit 110 may cause theone or more navigational responses based on the analysis performed atstep 720 and the techniques as described above in connection with FIG.4. Processing unit 110 may also use data derived from execution ofvelocity and acceleration module 406 to cause the one or morenavigational responses. In some embodiments, processing unit 110 maycause the one or more navigational responses based on a relativeposition, relative velocity, and/or relative acceleration betweenvehicle 200 and an object detected within any of the first, second, andthird plurality of images. Multiple navigational responses may occursimultaneously, in sequence, or any combination thereof.

Sparse Road Model for Autonomous Vehicle Navigation

In some embodiments, the disclosed systems and methods may use a sparsemap for autonomous vehicle navigation. For example, the sparse map mayprovide sufficient information for navigating an autonomous vehiclewithout storing and/or updating a large quantity of data. As discussedbelow in further detail, an autonomous vehicle may use the sparse map tonavigate one or more roads based on one or more stored trajectories.

Sparse Map for Autonomous Vehicle Navigation

In some embodiments, the disclosed systems and methods may use a sparsemap for autonomous vehicle navigation. For example, the sparse map mayprovide sufficient information for navigation without requiringexcessive data storage or data transfer rates. As discussed below infurther detail, a vehicle (which may be an autonomous vehicle) may usethe sparse map to navigate one or more roads. For example, in someembodiments, the sparse map may include data related to a road andpotentially landmarks along the road that may be sufficient for vehiclenavigation, but which also exhibit small data footprints. For example,the sparse data maps described in detail below may require significantlyless storage space and data transfer bandwidth as compared with digitalmaps including detailed map information, such as image data collectedalong a road. For example, rather than storing detailed representationsof a road segment, the sparse data map may store three dimensionalpolynomial representations of preferred vehicle paths along a road.These paths may require very little data storage space. Further, in thedescribed sparse data maps, landmarks may be identified and included inthe sparse map road model to aid in navigation. These landmarks may belocated at any spacing suitable for enabling vehicle navigation, but insome cases, such landmarks need not be identified and included in themodel at high densities and short spacings. Rather, in some cases,navigation may be possible based on landmarks that are spaced apart byat least 50 meters, at least 100 meters, at least 500 meters, at least 1kilometer, or at least 2 kilometers. As will be discussed in more detailin other sections, the sparse map may be generated based on datacollected or measured by vehicles equipped with various sensors anddevices, such as image capture devices, Global Positioning Systemsensors, motion sensors, etc., as the vehicles travel along roadways. Insome cases, the sparse map may be generated based on data collectedduring multiple drives of one or more vehicles along a particularroadway.

Consistent with disclosed embodiments, an autonomous vehicle system mayuse a sparse map for navigation. At the core of the sparse maps, one ormore three-dimensional contours may represent predetermined trajectoriesthat autonomous vehicles may traverse as they move along associated roadsegments. The sparse maps may also include data representing one or moreroad features. Such road features may include recognized landmarks, roadsignature profiles, and any other road-related features useful innavigating a vehicle. The sparse maps may enable autonomous navigationof a vehicle based on relatively small amounts of data included in thesparse map. For example, rather than including detailed representationsof a road, such as road edges, road curvature, images associated withroad segments, or data detailing other physical features associated witha road segment, the disclosed embodiments of the sparse map may requirerelatively little storage space (and relatively little bandwidth whenportions of the sparse map are transferred to a vehicle), but may stilladequately provide for autonomous vehicle navigation. The small datafootprint of the disclosed sparse maps, discussed in further detailbelow, may be achieved in some embodiments by storing representations ofroad-related elements that require small amounts of data, but stillenable autonomous navigation. For example, rather than storing detailedrepresentations of various aspects of a road, the disclosed sparse mapsmay store polynomial representations of one or more trajectories that avehicle may follow along the road. Thus, rather than storing (or havingto transfer) details regarding the physical nature of the road to enablenavigation along the road, using the disclosed sparse maps, a vehiclemay be navigated along a particular road segment without, in some cases,having to interpret physical aspects of the road, but rather, byaligning its path of travel with a trajectory (e.g., a polynomialspline) along the particular road segment. In this way, the vehicle maybe navigated based mainly upon the stored trajectory (e.g., a polynomialspline) that may require much less storage space than an approachinvolving storage of roadway images, road parameters, road layout, etc.

In addition to the stored polynomial representations of trajectoriesalong a road segment, the disclosed sparse maps may also include smalldata objects that may represent a road feature. In some embodiments, thesmall data objects may include digital signatures, which are derivedfrom a digital image (or a digital signal) that was obtained by a sensor(e.g., a camera or other sensor, such as a suspension sensor) onboard avehicle traveling along the road segment. The digital signature may havea reduced size relative to the signal that was acquired by the sensor.In some embodiments, the digital signature may be created to becompatible with a classifier function that is configured to detect andto identify the road feature from the signal that is acquired by thesensor, for example during a subsequent drive. In some embodiments, adigital signature may be created such that it has a footprint that is assmall as possible, while retaining the ability to correlate or match theroad feature with the stored signature based on an image (or a digitalsignal generated by a sensor, if the stored signature is not based on animage and/or includes other data) of the road feature that is capturedby a camera onboard a vehicle traveling along the same road segment at asubsequent time. In some embodiments, a size of the data objects may befurther associated with a uniqueness of the road feature. For example,for a road feature that is detectable by a camera onboard a vehicle, andwhere the camera system onboard the vehicle is coupled to a classifierwhich is capable of distinguishing the image data corresponding to thatroad feature as being associated with a particular type of road feature,for example, a road sign, and where such a road sign is locally uniquein that area (e.g., there is no identical road sign or road sign of thesame type nearby), it may be sufficient to store data indicating thetype of the road feature and its location.

As will be discussed in further detail below, road features (e.g.,landmarks along a road segment) may be stored as small data objects thatmay represent a road feature in relatively few bytes, while at the sametime providing sufficient information for recognizing and using such afeature for navigation. In just one example, a road sign may beidentified as a recognized landmark on which navigation of a vehicle maybe based. A representation of the road sign may be stored in the sparsemap to include, e.g., a few bytes of data indicating a type of landmark(e.g., a stop sign) and a few bytes of data indicating a location of thelandmark. Navigating based on such data-light representations of thelandmarks (e.g., using representations sufficient for locating,recognizing, and navigating based upon the landmarks) may provide adesired level of navigational functionality associated with sparse mapswithout significantly increasing the data overhead associated with thesparse maps. This lean representation of landmarks (and other roadfeatures) may take advantage of the sensors and processors includedonboard such vehicles that are configured to detect, identify, and/orclassify certain road features. When, for example, a sign or even aparticular type of a sign is locally unique (e.g., when there is noother sign or no other sign of the same type) in a given area, thesparse map may use data indicating a type of a landmark (a sign or aspecific type of sign), and during navigation (e.g., autonomousnavigation) when a camera onboard an autonomous vehicle captures animage of the area including a sign (or of a specific type of sign), theprocessor may process the image, detect the sign (if indeed present inthe image), classify it as a sign (or as a specific type of sign), andcorrelate its location with the location of the sign as stored in thesparse map.

In some embodiments, an autonomous vehicle may include a vehicle bodyand a processor configured to receive data included in a sparse map andgenerate navigational instructions for navigating the vehicle along aroad segment based on the data in the sparse map.

FIG. 8 shows a sparse map 800 that vehicle 200 (which may be anautonomous vehicle) may access for providing autonomous vehiclenavigation. Sparse map 800 may be stored in a memory, such as memory 140or 150. Such memory devices may include any types of non-transitorystorage devices or computer-readable media. For example, in someembodiments, memory 140 or 150 may include hard drives, compact discs,flash memory, magnetic based memory devices, optical based memorydevices, etc. In some embodiments, sparse map 800 may be stored in adatabase (e.g., map database 160) that may be stored in memory 140 or150, or other types of storage devices.

In some embodiments, sparse map 800 may be stored on a storage device ora non-transitory computer-readable medium provided onboard vehicle 200(e.g., a storage device included in a navigation system onboard vehicle200). A processor (e.g., processing unit 110) provided on vehicle 200may access sparse map 800 stored in the storage device orcomputer-readable medium provided onboard vehicle 200 in order togenerate navigational instructions for guiding the autonomous vehicle200 as it traverses a road segment.

Sparse map 800 need not be stored locally with respect to a vehicle,however. In some embodiments, sparse map 800 may be stored on a storagedevice or computer-readable medium provided on a remote server thatcommunicates with vehicle 200 or a device associated with vehicle 200. Aprocessor (e.g., processing unit 110) provided on vehicle 200 mayreceive data included in sparse map 800 from the remove server and mayexecute the data for guiding the autonomous driving of vehicle 200. Insuch embodiments, sparse map 800 may be made accessible to a pluralityof vehicles traversing various road segments (e.g., tens, hundreds,thousands, or millions of vehicles, etc.). It should be noted also thatsparse map 800 may include multiple sub-maps. For example, in someembodiments, sparse map 800 may include hundreds, thousands, millions,or more, of sub-maps that can be used in navigating a vehicle. Suchsub-maps may be referred to as local maps, and a vehicle traveling alonga roadway may access any number of local maps relevant to a location inwhich the vehicle is traveling. The local map sections of sparse map 800may be stored with a Global Navigation Satellite System (GNSS) key as anindex to the database of sparse map 800. Thus, while computation ofsteering angles for navigating a host vehicle in the present system maybe performed without reliance upon a GNSS position of the host vehicle,road features, or landmarks, such GNSS information may be used forretrieval of relevant local maps.

Collection of data and generation of sparse map 800 is covered in detailin other sections. In general, however, sparse map 800 may be generatedbased on data collected from one or more vehicles as they travel alongroadways. For example, using sensors aboard the one or more vehicles(e.g., cameras, speedometers, GPS, accelerometers, etc.), thetrajectories that the one or more vehicles travel along a roadway may berecorded, and the polynomial representation of a preferred trajectoryfor vehicles making subsequent trips along the roadway may be determinedbased on the collected trajectories travelled by the one or morevehicles. Similarly, data collected by the one or more vehicles may aidin identifying potential landmarks along a particular roadway. Datacollected from traversing vehicles may also be used to identify roadprofile information, such as road width profiles, road roughnessprofiles, traffic line spacing profiles, etc. Using the collectedinformation, sparse map 800 may be generated and distributed (e.g., forlocal storage or via on-the-fly data transmission) for use in navigatingone or more autonomous vehicles. Map generation may not end upon initialgeneration of the map, however. As will be discussed in greater detailin other sections, sparse map 800 may be continuously or periodicallyupdated based on data collected from vehicles as those vehicles continueto traverse roadways included in sparse map 800.

Data recorded in sparse map 800 may include position information basedon Global Positioning System (GPS) data. For example, locationinformation may be included in sparse map 800 for various map elements,including, for example, landmark locations, road profile locations, etc.Locations for map elements included in sparse map 800 may be obtainedusing GPS data collected from vehicles traversing a roadway. Forexample, a vehicle passing an identified landmark may determine alocation of the identified landmark using GPS position informationassociated with the vehicle and a determination of a location of theidentified landmark relative to the vehicle (e.g., based on imageanalysis of data collected from one or more cameras on board thevehicle). Such location determinations of an identified landmark (or anyother feature included in sparse map 800) may be repeated as additionalvehicles pass the location of the identified landmark. Some or all ofthe additional location determinations can be used to refine thelocation information stored in sparse map 800 relative to the identifiedlandmark. For example, in some embodiments, multiple positionmeasurements relative to a particular feature stored in sparse map 800may be averaged together. Any other mathematical operations, however,may also be used to refine a stored location of a map element based on aplurality of determined locations for the map element.

The sparse map of the disclosed embodiments may enable autonomousnavigation of a vehicle using relatively small amounts of stored data.In some embodiments, sparse map 800 may have a data density (e.g.,including data representing the target trajectories, landmarks, and anyother stored road features) of less than 2 MB per kilometer of roads,less than 1 MB per kilometer of roads, less than 500 kB per kilometer ofroads, or less than 100 kB per kilometer of roads. In some embodiments,the data density of sparse map 800 may be less than 10 kB per kilometerof roads or even less than 2 kB per kilometer of roads (e.g., 1.6 kB perkilometer), or no more than 10 kB per kilometer of roads, or no morethan 20 kB per kilometer of roads. In some embodiments, most if not allof the roadways of the United States may be navigated autonomously usinga sparse map having a total of 4 GB or less of data. These data densityvalues may represent an average over an entire sparse map 800, over alocal map within sparse map 800, and/or over a particular road segmentwithin sparse map 800.

As noted, sparse map 800 may include representations of a plurality oftarget trajectories 810 for guiding autonomous driving or navigationalong a road segment. Such target trajectories may be stored asthree-dimensional splines. The target trajectories stored in sparse map800 may be determined based on two or more reconstructed trajectories ofprior traversals of vehicles along a particular road segment. A roadsegment may be associated with a single target trajectory or multipletarget trajectories. For example, on a two lane road, a first targettrajectory may be stored to represent an intended path of travel alongthe road in a first direction, and a second target trajectory may bestored to represent an intended path of travel along the road in anotherdirection (e.g., opposite to the first direction). Additional targettrajectories may be stored with respect to a particular road segment.For example, on a multi-lane road one or more target trajectories may bestored representing intended paths of travel for vehicles in one or morelanes associated with the multi-lane road. In some embodiments, eachlane of a multi-lane road may be associated with its own targettrajectory. In other embodiments, there may be fewer target trajectoriesstored than lanes present on a multi-lane road. In such cases, a vehiclenavigating the multi-lane road may use any of the stored targettrajectories to guides its navigation by taking into account an amountof lane offset from a lane for which a target trajectory is stored(e.g., if a vehicle is traveling in the left most lane of a three lanehighway, and a target trajectory is stored only for the middle lane ofthe highway, the vehicle may navigate using the target trajectory of themiddle lane by accounting for the amount of lane offset between themiddle lane and the left-most lane when generating navigationalinstructions).

In some embodiments, the target trajectory may represent an ideal paththat a vehicle should take as the vehicle travels. The target trajectorymay be located, for example, at an approximate center of a lane oftravel. In other cases, the target trajectory may be located elsewhererelative to a road segment. For example, a target trajectory mayapproximately coincide with a center of a road, an edge of a road, or anedge of a lane, etc. In such cases, navigation based on the targettrajectory may include a determined amount of offset to be maintainedrelative to the location of the target trajectory. Moreover, in someembodiments, the determined amount of offset to be maintained relativeto the location of the target trajectory may differ based on a type ofvehicle (e.g., a passenger vehicle including two axles may have adifferent offset from a truck including more than two axles along atleast a portion of the target trajectory).

Sparse map 800 may also include data relating to a plurality ofpredetermined landmarks 820 associated with particular road segments,local maps, etc. As discussed in detail in other sections, theselandmarks may be used in navigation of the autonomous vehicle. Forexample, in some embodiments, the landmarks may be used to determine acurrent position of the vehicle relative to a stored target trajectory.With this position information, the autonomous vehicle may be able toadjust a heading direction to match a direction of the target trajectoryat the determined location.

The plurality of landmarks 820 may be identified and stored in sparsemap 800 at any suitable spacing. In some embodiments, landmarks may bestored at relatively high densities (e.g., every few meters or more). Insome embodiments, however, significantly larger landmark spacing valuesmay be employed. For example, in sparse map 800, identified (orrecognized) landmarks may be spaced apart by 10 meters, 20 meters, 50meters, 100 meters, 1 kilometer, or 2 kilometers. In some cases, theidentified landmarks may be located at distances of even more than 2kilometers apart. Between landmarks, and therefore betweendeterminations of vehicle position relative to a target trajectory, thevehicle may navigate based on dead reckoning in which it uses sensors todetermine its ego motion and estimate its position relative to thetarget trajectory. Because errors may accumulate during navigation bydead reckoning, over time the position determinations relative to thetarget trajectory may become increasingly less accurate. The vehicle mayuse landmarks occurring in sparse map 800 (and their known locations) toremove the dead reckoning-induced errors in position determination. Inthis way, the identified landmarks included in sparse map 800 may serveas navigational anchors from which an accurate position of the vehiclerelative to a target trajectory may be determined. Because a certainamount of error may be acceptable in position location, an identifiedlandmark need not always be available to an autonomous vehicle. Rather,suitable navigation may be possible even based on landmark spacings, asnoted above, of 10 meters, 20 meters, 50 meters, 100 meters, 500 meters,1 kilometer, 2 kilometers, or more. In some embodiments, a density of 1identified landmark every 1 km of road may be sufficient to maintain alongitudinal position determination accuracy within 1 m. Thus, not everypotential landmark appearing along a road segment need be stored insparse map 800.

In addition to target trajectories and identified landmarks, sparse map800 may include information relating to various other road features. Forexample, FIG. 9A illustrates a representation of curves along aparticular road segment that may be stored in sparse map 800. In someembodiments, a single lane of a road may be modeled by athree-dimensional polynomial description of left and right sides of theroad. Such polynomials representing left and right sides of a singlelane are shown in FIG. 9A. Regardless of how many lanes a road may have,the road may be represented using polynomials in a way similar to thatillustrated in FIG. 9A. For example, left and right sides of amulti-lane road may be represented by polynomials similar to those shownin FIG. 9A, and intermediate lane markings included on a multi-lane road(e.g., dashed markings representing lane boundaries, solid yellow linesrepresenting boundaries between lanes traveling in different directions,etc.) may also be represented using polynomials such as those shown inFIG. 9A.

As shown in FIG. 9A, a lane 900 may be represented using polynomials(e.g., a first order, second order, third order, or any suitable orderpolynomials). For illustration, lane 900 is shown as a two-dimensionallane and the polynomials are shown as two-dimensional polynomials. Lane900 includes a left side 910 and a right side 920. In some embodiments,more than one polynomial may be used to represent a location of eachside of the road or lane boundary. For example, each of left side 910and right side 920 may be represented by a plurality of polynomials ofany suitable length. In some cases, the polynomials may have a length ofabout 100 m, although other lengths greater than or less than 100 m mayalso be used. Additionally, the polynomials can overlap with one anotherin order to facilitate seamless transitions in navigating based onsubsequently encountered polynomials as a host vehicle travels along aroadway. For example, each of left side 910 and right side 920 may berepresented by a plurality of third order polynomials separated intosegments of about 100 meters in length (an example of the firstpredetermined range), and overlapping each other by about 50 meters. Thepolynomials representing the left side 910 and the right side 920 may ormay not have the same order. For example, in some embodiments, somepolynomials may be second order polynomials, some may be third orderpolynomials, and some may be fourth order polynomials.

In the example shown in FIG. 9A, left side 910 of lane 900 isrepresented by two groups of third order polynomials. The first groupincludes polynomial segments 911, 912, and 913. The second groupincludes polynomial segments 914, 915, and 916. The two groups, whilesubstantially parallel to each other, follow the locations of theirrespective sides of the road. Polynomial segments 911-916 have a lengthof about 100 meters and overlap adjacent segments in the series by about50 meters. As noted previously, however, polynomials of differentlengths and different overlap amounts may also be used. For example, thepolynomials may have lengths of 500 m, 1 km, or more, and the overlapamount may vary from 0 to 50 m, 50 m to 100 m, or greater than 100 m.Additionally, while FIG. 9A is shown as representing polynomialsextending in 2D space (e.g., on the surface of the paper), it is to beunderstood that these polynomials may represent curves extending inthree dimensions (e.g., including a height component) to representelevation changes in a road segment in addition to X-Y curvature.

Returning to the target trajectories of sparse map 800, FIG. 9B shows athree-dimensional polynomial representing a target trajectory for avehicle traveling along a particular road segment. The target trajectoryrepresents not only the X-Y path that a host vehicle should travel alonga particular road segment, but also the elevation change that the hostvehicle will experience when traveling along the road segment. Thus,each target trajectory in sparse map 800 may be represented by one ormore three-dimensional polynomials, like the three-dimensionalpolynomial 950 shown in FIG. 9B. Sparse map 800 may include a pluralityof trajectories (e.g., millions or billions or more to representtrajectories of vehicles along various road segments along roadwaysthroughout the world). In some embodiments, each target trajectory maycorrespond to a spline connecting three-dimensional polynomial segments.

Regarding the data footprint of polynomial curves stored in sparse map800, in some embodiments, each third degree polynomial may berepresented by four parameters, each requiring four bytes of data.Suitable representations may be obtained with third degree polynomialsrequiring about 192 bytes of data for every 100 m. This translates toapproximately 200 kB per hour in data usage/transfer requirements for ahost vehicle traveling approximately 100 km/hr.

Sparse map 800 may describe the lanes network using a combination ofgeometry descriptors and meta-data. The geometry may be described bypolynomials or splines as described above. The meta-data may describethe number of lanes, special characteristics (such as a car pool lane),and possibly other sparse labels. The total footprint of such indicatorsmay be negligible.

As previously noted, sparse map 800 may include a plurality ofpredetermined landmarks associated with a road segment. Rather thanstoring actual images of the landmarks and relying, for example, onimage recognition analysis based on captured images and stored images,each landmark in sparse map 800 may be represented and recognized usingless data than a stored, actual image would require. Data representinglandmarks may include sufficient information for describing oridentifying the landmarks along a road. Storing data describingcharacteristics of landmarks, rather than the actual images oflandmarks, may reduce the size of sparse map 800.

FIG. 10 illustrates examples of types of landmarks that may berepresented in sparse map 800. The landmarks may include any visible andidentifiable objects along a road segment. The landmarks may be selectedsuch that they are fixed and do not change often with respect to theirlocations and/or content. The landmarks included in sparse map 800 maybe useful in determining a location of vehicle 200 with respect to atarget trajectory as the vehicle traverses a particular road segment.Examples of landmarks may include traffic signs, directional signs,general signs (e.g., rectangular signs), roadside fixtures (e.g.,lampposts, reflectors, etc.), and any other suitable category. In someembodiments, lane marks on the road, may also be included as landmarksin sparse map 800.

Examples of landmarks shown in FIG. 10 include traffic signs,directional signs, roadside fixtures, and general signs. Traffic signsmay include, for example, speed limit signs (e.g., speed limit sign1000), yield signs (e.g., yield sign 1005), route number signs (e.g.,route number sign 1010), traffic light signs (e.g., traffic light sign1015), stop signs (e.g., stop sign 1020). Directional signs may includea sign that includes one or more arrows indicating one or moredirections to different places. For example, directional signs mayinclude a highway sign 1025 having arrows for directing vehicles todifferent roads or places, an exit sign 1030 having an arrow directingvehicles off a road, etc.

General signs may be unrelated to traffic. For example, general signsmay include billboards used for advertisement, or a welcome boardadjacent a border between two countries, states, counties, cities, ortowns. FIG. 10 shows a general sign 1040 (“Joe's Restaurant”). Althoughgeneral sign 1040 may have a rectangular shape, as shown in FIG. 10,general sign 1040 may have other shapes, such as square, circle,triangle, etc.

Landmarks may also include roadside fixtures. Roadside fixtures may beobjects that are not signs, and may not be related to traffic ordirections. For example, roadside fixtures may include lampposts (e.g.,lamppost 1035), power line posts, traffic light posts, etc.

Landmarks may also include beacons that may be specifically designed forusage in an autonomous vehicle navigation system. For example, suchbeacons may include stand-alone structures placed at predeterminedintervals to aid in navigating a host vehicle. Such beacons may alsoinclude visual/graphical information added to existing road signs (e.g.,icons, emblems, bar codes, etc.) that may be identified or recognized bya vehicle traveling along a road segment. Such beacons may also includeelectronic components. In such embodiments, electronic beacons (e.g.,RFID tags, etc.) may be used to transmit non-visual information to ahost vehicle. Such information may include, for example, landmarkidentification and/or landmark location information that a host vehiclemay use in determining its position along a target trajectory.

In some embodiments, the landmarks included in sparse map 800 may berepresented by a data object of a predetermined size. The datarepresenting a landmark may include any suitable parameters foridentifying a particular landmark. For example, in some embodiments,landmarks stored in sparse map 800 may include parameters such as aphysical size of the landmark (e.g., to support estimation of distanceto the landmark based on a known size/scale), a distance to a previouslandmark, lateral offset, height, a type code (e.g., a landmark typewhat type of directional sign, traffic sign, etc.), a GPS coordinate(e.g., to support global localization), and any other suitableparameters. Each parameter may be associated with a data size. Forexample, a landmark size may be stored using 8 bytes of data. A distanceto a previous landmark, a lateral offset, and height may be specifiedusing 12 bytes of data. A type code associated with a landmark such as adirectional sign or a traffic sign may require about 2 bytes of data.For general signs, an image signature enabling identification of thegeneral sign may be stored using 50 bytes of data storage. The landmarkGPS position may be associated with 16 bytes of data storage. These datasizes for each parameter are examples only, and other data sizes mayalso be used.

Representing landmarks in sparse map 800 in this manner may offer a leansolution for efficiently representing landmarks in the database. In someembodiments, signs may be referred to as semantic signs and non-semanticsigns. A semantic sign may include any class of signs for which there'sa standardized meaning (e.g., speed limit signs, warning signs,directional signs, etc.). A non-semantic sign may include any sign thatis not associated with a standardized meaning (e.g., general advertisingsigns, signs identifying business establishments, etc.). For example,each semantic sign may be represented with 38 bytes of data (e.g., 8bytes for size; 12 bytes for distance to previous landmark, lateraloffset, and height; 2 bytes for a type code; and 16 bytes for GPScoordinates). Sparse map 800 may use a tag system to represent landmarktypes. In some cases, each traffic sign or directional sign may beassociated with its own tag, which may be stored in the database as partof the landmark identification. For example, the database may include onthe order of 1000 different tags to represent various traffic signs andon the order of about 10000 different tags to represent directionalsigns. Of course, any suitable number of tags may be used, andadditional tags may be created as needed. General purpose signs may berepresented in some embodiments using less than about 100 bytes (e.g.,about 86 bytes including 8 bytes for size; 12 bytes for distance toprevious landmark, lateral offset, and height; 50 bytes for an imagesignature; and 16 bytes for GPS coordinates).

Thus, for semantic road signs not requiring an image signature, the datadensity impact to sparse map 800, even at relatively high landmarkdensities of about 1 per 50 m, may be on the order of about 760 bytesper kilometer (e.g., 20 landmarks per km×38 bytes per landmark=760bytes). Even for general purpose signs including an image signaturecomponent, the data density impact is about 1.72 kB per km (e.g., 20landmarks per km×86 bytes per landmark=1,720 bytes). For semantic roadsigns, this equates to about 76 kB per hour of data usage for a vehicletraveling 100 km/hr. For general purpose signs, this equates to about170 kB per hour for a vehicle traveling 100 km/hr.

In some embodiments, a generally rectangular object, such as arectangular sign, may be represented in sparse map 800 by no more than100 byte of data. The representation of the generally rectangular object(e.g., general sign 1040) in sparse map 800 may include a condensedimage signature (e.g., condensed image signature 1045) associated withthe generally rectangular object. This condensed image signature may beused, for example, to aid in identification of a general purpose sign,for example, as a recognized landmark. Such a condensed image signature(e.g., image information derived from actual image data representing anobject) may avoid a need for storage of an actual image of an object ora need for comparative image analysis performed on actual images inorder to recognize landmarks.

Referring to FIG. 10, sparse map 800 may include or store a condensedimage signature 1045 associated with a general sign 1040, rather than anactual image of general sign 1040. For example, after an image capturedevice (e.g., image capture device 122, 124, or 126) captures an imageof general sign 1040, a processor (e.g., image processor 190 or anyother processor that can process images either aboard or remotelylocated relative to a host vehicle) may perform an image analysis toextract/create condensed image signature 1045 that includes a uniquesignature or pattern associated with general sign 1040. In oneembodiment, condensed image signature 1045 may include a shape, colorpattern, a brightness pattern, or any other feature that may beextracted from the image of general sign 1040 for describing generalsign 1040. For example, in FIG. 10, the circles, triangles, and starsshown in condensed image signature 1045 may represent areas of differentcolors. The pattern represented by the circles, triangles, and stars maybe stored in sparse map 800, e.g., within the 50 bytes designated toinclude an image signature. Notably, the circles, triangles, and starsare not necessarily meant to indicate that such shapes are stored aspart of the image signature. Rather, these shapes are meant toconceptually represent recognizable areas having discernible colordifferences, textual areas, graphical shapes, or other variations incharacteristics that may be associated with a general purpose sign. Suchcondensed image signatures can be used to identify a landmark in theform of a general sign. For example, the condensed image signature canbe used to perform a same-not-same analysis based on a comparison of astored condensed image signature with image data captured, for example,using a camera onboard an autonomous vehicle.

Returning to the target trajectories a host vehicle may use to navigatea particular road segment, FIG. 11A shows polynomial representationstrajectories capturing during a process of building or maintainingsparse map 800. A polynomial representation of a target trajectoryincluded in sparse map 800 may be determined based on two or morereconstructed trajectories of prior traversals of vehicles along thesame road segment. In some embodiments, the polynomial representation ofthe target trajectory included in sparse map 800 may be an aggregationof two or more reconstructed trajectories of prior traversals ofvehicles along the same road segment. In some embodiments, thepolynomial representation of the target trajectory included in sparsemap 800 may be an average of the two or more reconstructed trajectoriesof prior traversals of vehicles along the same road segment. Othermathematical operations may also be used to construct a targettrajectory along a road path based on reconstructed trajectoriescollected from vehicles traversing along a road segment.

As shown in FIG. 11A, a road segment 1100 may be travelled by a numberof vehicles 200 at different times. Each vehicle 200 may collect datarelating to a path that it took along the road segment. The pathtraveled by a particular vehicle may be determined based on camera data,accelerometer information, speed sensor information, and/or GPSinformation, among other potential sources. Such data may be used toreconstruct trajectories of vehicles traveling along the road segment,and based on these reconstructed trajectories, a target trajectory (ormultiple target trajectories) may be determined for the particular roadsegment. Such target trajectories may represent a preferred path of ahost vehicle (e.g., guided by an autonomous navigation system) as ittravels along the road segment.

In the example shown in FIG. 11A, a first reconstructed trajectory 1101may be determined based on data received from a first vehicle traversingroad segment 1100 at a first time period (e.g., day 1), a secondreconstructed trajectory 1102 may be obtained from a second vehicletraversing road segment 1100 at a second time period (e.g., day 2), anda third reconstructed trajectory 1103 may be obtained from a thirdvehicle traversing road segment 1100 at a third time period (e.g., day3). Each trajectory 1101, 1102, and 1103 may be represented by apolynomial, such as a three-dimensional polynomial. It should be notedthat in some embodiments, any of the reconstructed trajectories may beassembled onboard the vehicles traversing road segment 1100.

Additionally, or alternatively, such reconstructed trajectories may bedetermined on a server side based on information received from vehiclestraversing road segment 1100. For example, in some embodiments, vehicles200 may transmit data to one or more servers relating to their motionalong road segment 1100 (e.g., steering angle, heading, time, position,speed, sensed road geometry, and/or sensed landmarks, among things). Theserver may reconstruct trajectories for vehicles 200 based on thereceived data. The server may also generate a target trajectory forguiding navigation of autonomous vehicle that will travel along the sameroad segment 1100 at a later time based on the first, second, and thirdtrajectories 1101, 1102, and 1103. While a target trajectory may beassociated with a single prior traversal of a road segment, in someembodiments, each target trajectory included in sparse map 800 may bedetermined based on two or more reconstructed trajectories of vehiclestraversing the same road segment. In FIG. 11A, the target trajectory isrepresented by 1110. In some embodiments, the target trajectory 1110 maybe generated based on an average of the first, second, and thirdtrajectories 1101, 1102, and 1103. In some embodiments, the targettrajectory 1110 included in sparse map 800 may be an aggregation (e.g.,a weighted combination) of two or more reconstructed trajectories.

FIGS. 11B and 11C further illustrate the concept of target trajectoriesassociated with road segments present within a geographic region 1111.As shown in FIG. 11B, a first road segment 1120 within geographic region1111 may include a multilane road, which includes two lanes 1122designated for vehicle travel in a first direction and two additionallanes 1124 designated for vehicle travel in a second direction oppositeto the first direction. Lanes 1122 and lanes 1124 may be separated by adouble yellow line 1123. Geographic region 1111 may also include abranching road segment 1130 that intersects with road segment 1120. Roadsegment 1130 may include a two-lane road, each lane being designated fora different direction of travel. Geographic region 1111 may also includeother road features, such as a stop line 1132, a stop sign 1134, a speedlimit sign 1136, and a hazard sign 1138.

As shown in FIG. 11C, sparse map 800 may include a local map 1140including a road model for assisting with autonomous navigation ofvehicles within geographic region 1111. For example, local map 1140 mayinclude target trajectories for one or more lanes associated with roadsegments 1120 and/or 1130 within geographic region 1111. For example,local map 1140 may include target trajectories 1141 and/or 1142 that anautonomous vehicle may access or rely upon when traversing lanes 1122.Similarly, local map 1140 may include target trajectories 1143 and/or1144 that an autonomous vehicle may access or rely upon when traversinglanes 1124. Further, local map 1140 may include target trajectories 1145and/or 1146 that an autonomous vehicle may access or rely upon whentraversing road segment 1130. Target trajectory 1147 represents apreferred path an autonomous vehicle should follow when transitioningfrom lanes 1120 (and specifically, relative to target trajectory 1141associated with a right-most lane of lanes 1120) to road segment 1130(and specifically, relative to a target trajectory 1145 associated witha first side of road segment 1130. Similarly, target trajectory 1148represents a preferred path an autonomous vehicle should follow whentransitioning from road segment 1130 (and specifically, relative totarget trajectory 1146) to a portion of road segment 1124 (andspecifically, as shown, relative to a target trajectory 1143 associatedwith a left lane of lanes 1124.

Sparse map 800 may also include representations of other road-relatedfeatures associated with geographic region 1111. For example, sparse map800 may also include representations of one or more landmarks identifiedin geographic region 1111. Such landmarks may include a first landmark1150 associated with stop line 1132, a second landmark 1152 associatedwith stop sign 1134, a third landmark associated with speed limit sign1154, and a fourth landmark 1156 associated with hazard sign 1138. Suchlandmarks may be used, for example, to assist an autonomous vehicle indetermining its current location relative to any of the shown targettrajectories, such that the vehicle may adjust its heading to match adirection of the target trajectory at the determined location.

In some embodiments, sparse may 800 may also include road signatureprofiles. Such road signature profiles may be associated with anydiscernible/measurable variation in at least one parameter associatedwith a road. For example, in some cases, such profiles may be associatedwith variations in surface roughness of a particular road segment,variations in road width over a particular road segment, variations indistances between dashed lines painted along a particular road segment,variations in road curvature along a particular road segment, etc. FIG.11D shows an example of a road signature profile 1160. While profile1160 may represent any of the parameters mentioned above, or others, inone example, profile 1160 may represent a measure of road surfaceroughness, as obtained, for example, by monitoring one or more sensorsproviding outputs indicative of an amount of suspension displacement asa vehicle travels a particular road segment. Alternatively, profile 1160may represent variation in road width, as determined based on image dataobtained via a camera onboard a vehicle traveling a particular roadsegment. Such profiles may be useful, for example, in determining aparticular location of an autonomous vehicle relative to a particulartarget trajectory. That is, as it traverses a road segment, anautonomous vehicle may measure a profile associated with one or moreparameters associated with the road segment. If the measured profile canbe correlated/matched with a predetermined profile that plots theparameter variation with respect to position along the road segment,then the measured and predetermined profiles may be used (e.g., byoverlaying corresponding sections of the measured and predeterminedprofiles) in order to determine a current position along the roadsegment and, therefore, a current position relative to a targettrajectory for the road segment.

In some embodiments, sparse map 800 may include different trajectoriesbased on different characteristics associated with a user of autonomousvehicles, environmental conditions, and/or other parameters relating todriving. For example, in some embodiments, different trajectories may begenerated based on different user preferences and/or profiles. Sparsemap 800 including such different trajectories may be provided todifferent autonomous vehicles of different users. For example, someusers may prefer to avoid toll roads, while others may prefer to takethe shortest or fastest routes, regardless of whether there is a tollroad on the route. The disclosed systems may generate different sparsemaps with different trajectories based on such different userpreferences or profiles. As another example, some users may prefer totravel in a fast moving lane, while others may prefer to maintain aposition in the central lane at all times.

Different trajectories may be generated and included in sparse map 800based on different environmental conditions, such as day and night,snow, rain, fog, etc. Autonomous vehicles driving under differentenvironmental conditions may be provided with sparse map 800 generatedbased on such different environmental conditions. In some embodiments,cameras provided on autonomous vehicles may detect the environmentalconditions, and may provide such information back to a server thatgenerates and provides sparse maps. For example, the server may generateor update an already generated sparse map 800 to include trajectoriesthat may be more suitable or safer for autonomous driving under thedetected environmental conditions. The update of sparse map 800 based onenvironmental conditions may be performed dynamically as the autonomousvehicles are traveling along roads.

Other different parameters relating to driving may also be used as abasis for generating and providing different sparse maps to differentautonomous vehicles. For example, when an autonomous vehicle istraveling at a high speed, turns may be tighter. Trajectories associatedwith specific lanes, rather than roads, may be included in sparse map800 such that the autonomous vehicle may maintain within a specific laneas it follows a specific trajectory. When an image captured by a cameraonboard the autonomous vehicle indicates that the vehicle has driftedoutside of the lane (e.g., crossed the lane mark), an action may betriggered within the vehicle to bring the vehicle back to the designatedlane according to the specific trajectory.

Constructing a Road Model for Autonomous Vehicle Navigation

In some embodiments, the disclosed systems and methods may construct aroad model for autonomous vehicle navigation. For example, the roadmodel may include crowd sourced data. The disclosed systems and methodsmay refine the crowd sourced data based on observed local conditions.Further, the disclosed systems and methods may determine a refinedtrajectory for an autonomous vehicle based on sensor information. Stillfurther, the disclosed systems and methods may identify landmarks foruse in the road model, as well refine the positions of the landmarks inthe road model. These systems and methods are disclosed in furtherdetail in the following sections.

Crowd Sourcing Data for Autonomous Vehicle Navigation

In some embodiments, the disclosed systems and methods may construct aroad model for autonomous vehicle navigation. For example, disclosedsystems and methods may use crowd sourced data for generation of anautonomous vehicle road model that one or more autonomous vehicles mayuse to navigate along a system of roads. By crowd sourcing, it meansthat data are received from various vehicles (e.g., autonomous vehicles)travelling on a road segment at different times and such data are usedto generate and/or update the road model. The model may, in turn, betransmitted to the vehicles or other vehicles later travelling along theroad segment for assisting autonomous vehicle navigation. The road modelmay include a plurality of target trajectories representing preferredtrajectories that autonomous vehicles should follow as they traverse aroad segment. The target trajectories may be the same as a reconstructedactual trajectory collected from a vehicle traversing a road segment,which may be transmitted from the vehicle to a server. In someembodiments, the target trajectories may be different from actualtrajectories that one or more vehicles previously took when traversing aroad segment. The target trajectories may be generated based on actualtrajectories (e.g., through averaging or any other suitable operation).

The vehicle trajectory data that a vehicle may upload to a server maycorrespond with the actual reconstructed trajectory for the vehicle, orit may correspond to a recommended trajectory, which may be based on orrelated to the actual reconstructed trajectory of the vehicle, but maydiffer from the actual reconstructed trajectory. For example, vehiclesmay modify their actual, reconstructed trajectories and submit (e.g.,recommend) to the server the modified actual trajectories. The roadmodel may use the recommended, modified trajectories as targettrajectories for autonomous navigation of other vehicles.

In addition to trajectory information, other information for potentialuse in building a sparse data map 800 may include information relatingto potential landmark candidates. For example, through crowd sourcing ofinformation, the disclosed systems and methods may identify potentiallandmarks in an environment and refine landmark positions. The landmarksmay be used by a navigation system of autonomous vehicles to determineand/or adjust the position of the vehicle along the target trajectories.

The reconstructed trajectories that a vehicle may generate as it travelsalong a road may be obtained by any suitable method. In someembodiments, the reconstructed trajectories may be developed bystitching together segments of motion for the vehicle, using, e.g., egomotion estimation (e.g., three dimensional translation and threedimensional rotation of the camera, and hence the body of the vehicle).The rotation and translation estimation may be determined based onanalysis of images captured by one or more image capture devices alongwith information from other sensors or devices, such as inertial sensorsand speed sensors. For example, the inertial sensors may include anaccelerometer or other suitable sensors configured to measure changes intranslation and/or rotation of the vehicle body. The vehicle may includea speed sensor that measures a speed of the vehicle.

In some embodiments, the ego motion of the camera (and hence the vehiclebody) may be estimated based on an optical flow analysis of the capturedimages. An optical flow analysis of a sequence of images identifiesmovement of pixels from the sequence of images, and based on theidentified movement, determines motions of the vehicle. The ego motionmay be integrated over time and along the road segment to reconstruct atrajectory associated with the road segment that the vehicle hasfollowed.

Data (e.g., reconstructed trajectories) collected by multiple vehiclesin multiple drives along a road segment at different times may be usedto construct the road model (e.g., including the target trajectories,etc.) included in sparse data map 800. Data collected by multiplevehicles in multiple drives along a road segment at different times mayalso be averaged to increase an accuracy of the model. In someembodiments, data regarding the road geometry and/or landmarks may bereceived from multiple vehicles that travel through the common roadsegment at different times. Such data received from different vehiclesmay be combined to generate the road model and/or to update the roadmodel.

The disclosed systems and methods may enable autonomous vehiclenavigation (e.g., steering control) with low footprint models, which maybe collected by the autonomous vehicles themselves without the aid ofexpensive surveying equipment. To support the autonomous navigation(e.g., steering applications), the road model may include the geometryof the road, its lane structure, and landmarks that may be used todetermine the location or position of vehicles along a trajectoryincluded in the model. Generation of the road model may be performed bya remote server that communicates with vehicles travelling on the roadand that receives data from the vehicles. The data may include senseddata, trajectories reconstructed based on the sensed data, and/orrecommended trajectories that may represent modified reconstructedtrajectories. The server may transmit the model back to the vehicles orother vehicles that later travel on the road to aid in autonomousnavigation.

The geometry of a reconstructed trajectory (and also a targettrajectory) along a road segment may be represented by a curve in threedimensional space, which may be a spline connecting three dimensionalpolynomials. The reconstructed trajectory curve may be determined fromanalysis of a video stream or a plurality of images captured by a camerainstalled on the vehicle. In some embodiments, a location is identifiedin each frame or image that is a few meters ahead of the currentposition of the vehicle. This location is where the vehicle is expectedto travel to in a predetermined time period. This operation may berepeated frame by frame, and at the same time, the vehicle may computethe camera's ego motion (rotation and translation). At each frame orimage, a short range model for the desired path is generated by thevehicle in a reference frame that is attached to the camera. The shortrange models may be stitched together to obtain a three dimensionalmodel of the road in some coordinate frame, which may be an arbitrary orpredetermined coordinate frame. The three dimensional model of the roadmay then be fitted by a spline, which may include or connect one or morepolynomials of suitable orders.

To conclude the short range road model at each frame, one or moredetection modules may be used. For example, a bottom-up lane detectionmodule may be used. The bottom-up lane detection module may be usefulwhen lane marks are drawn on the road. This module may look for edges inthe image and assembles them together to form the lane marks. A secondmodule may be used together with the bottom-up lane detection module.The second module is an end-to-end deep neural network, which may betrained to predict the correct short range path from an input image. Inboth modules, the road model may be detected in the image coordinateframe and transformed to a three dimensional space that may be virtuallyattached to the camera.

Although the reconstructed trajectory modeling method may introduce anaccumulation of errors due to the integration of ego motion over a longperiod of time, which may include a noise component, such errors may beinconsequential as the generated model may provide sufficient accuracyfor navigation over a local scale. In addition, it is possible to cancelthe integrated error by using external sources of information, such assatellite images or geodetic measurements. For example, the disclosedsystems and methods may use a GNSS receiver to cancel accumulatederrors. However, the GNSS positioning signals may not be alwaysavailable and accurate. The disclosed systems and methods may enable asteering application that depends weakly on the availability andaccuracy of GNSS positioning. In such systems, the usage of the GNSSsignals may be limited. For example, in some embodiments, the disclosedsystems may use the GNSS signals for database indexing purposes only.

In some embodiments, the range scale (e.g., local scale) that may berelevant for an autonomous vehicle navigation steering application maybe on the order of 50 meters, 100 meters, 200 meters, 300 meters, etc.Such distances may be used, as the geometrical road model is mainly usedfor two purposes: planning the trajectory ahead and localizing thevehicle on the road model. In some embodiments, the planning task mayuse the model over a typical range of 40 meters ahead (or any othersuitable distance ahead, such as 20 meters, 30 meters, 50 meters), whenthe control algorithm steers the vehicle according to a target pointlocated 1.3 seconds ahead (or any other time such as 1.5 seconds, 1.7seconds, 2 seconds, etc.). The localization task uses the road modelover a typical range of 60 meters behind the car (or any other suitabledistances, such as 50 meters, 100 meters, 150 meters, etc.), accordingto a method called “tail alignment” described in more detail in anothersection. The disclosed systems and methods may generate a geometricalmodel that has sufficient accuracy over particular range, such as 100meters, such that a planned trajectory will not deviate by more than,for example, 30 cm from the lane center.

As explained above, a three dimensional road model may be constructedfrom detecting short range sections and stitching them together. Thestitching may be enabled by computing a six degree ego motion model,using the videos and/or images captured by the camera, data from theinertial sensors that reflect the motions of the vehicle, and the hostvehicle velocity signal. The accumulated error may be small enough oversome local range scale, such as of the order of 100 meters. All this maybe completed in a single drive over a particular road segment.

In some embodiments, multiple drives may be used to average the resultedmodel, and to increase its accuracy further. The same car may travel thesame route multiple times, or multiple cars may send their collectedmodel data to a central server. In any case, a matching procedure may beperformed to identify overlapping models and to enable averaging inorder to generate target trajectories. The constructed model (e.g.,including the target trajectories) may be used for steering once aconvergence criterion is met. Subsequent drives may be used for furthermodel improvements and in order to accommodate infrastructure changes.

Sharing of driving experience (such as sensed data) between multiplecars becomes feasible if they are connected to a central server. Eachvehicle client may store a partial copy of a universal road model, whichmay be relevant for its current position. A bidirectional updateprocedure between the vehicles and the server may be performed by thevehicles and the server. The small footprint concept discussed aboveenables the disclosed systems and methods to perform the bidirectionalupdates using a very small bandwidth.

Information relating to potential landmarks may also be determined andforwarded to a central server. For example, the disclosed systems andmethods may determine one or more physical properties of a potentiallandmark based on one or more images that include the landmark. Thephysical properties may include a physical size (e.g., height, width) ofthe landmark, a distance from a vehicle to a landmark, a distancebetween the landmark to a previous landmark, the lateral position of thelandmark (e.g., the position of the landmark relative to the lane oftravel), the GPS coordinates of the landmark, a type of landmark,identification of text on the landmark, etc. For example, a vehicle mayanalyze one or more images captured by a camera to detect a potentiallandmark, such as a speed limit sign. The vehicle may determine adistance from the vehicle to the landmark based on the analysis of theone or more images. In some embodiments, the distance may be determinedbased on analysis of images of the landmark using a suitable imageanalysis method, such as a scaling method and/or an optical flow method.In some embodiments, the disclosed systems and methods may be configuredto determine a type or classification of a potential landmark. In casethe vehicle determines that a certain potential landmark corresponds toa predetermined type or classification stored in a sparse map, it may besufficient for the vehicle to communicate to the server an indication ofthe type or classification of the landmark, along with its location. Theserver may store such indications. At a later time, other vehicles maycapture an image of the landmark, process the image (e.g., using aclassifier), and compare the result from processing the image to theindication stored in the server with regard to the type of landmark.There may be various types of landmarks, and different types oflandmarks may be associated with different types of data to be uploadedto and stored in the server, different processing onboard the vehiclemay detects the landmark and communicate information about the landmarkto the server, and the system onboard the vehicle may receive thelandmark data from the server and use the landmark data for identifyinga landmark in autonomous navigation.

In some embodiments, multiple autonomous vehicles travelling on a roadsegment may communicate with a server. The vehicles (or clients) maygenerate a curve describing its drive (e.g., through ego motionintegration) in an arbitrary coordinate frame. The vehicles may detectlandmarks and locate them in the same frame. The vehicles may upload thecurve and the landmarks to the server. The server may collect data fromvehicles over multiple drives, and generate a unified road model. Theserver may distribute the model to clients (e.g., vehicles). The servermay continuously or periodically update the model when receiving newdata from the vehicles. For example, the server may process the new datato evaluate whether it includes information that should trigger anupdated, or creation of new data on the server. The server maydistribute the updated model or the updates to the vehicles forproviding autonomous vehicle navigation.

The server may use one or more criteria for determining whether new datareceived from the vehicles should trigger an update to the model ortrigger creation of new data. For example, when the new data indicatesthat a previously recognized landmark at a specific location no longerexists, or is replaced by another landmark, the server may determinethat the new data should trigger an update to the model. As anotherexample, when the new data indicates that a road segment has beenclosed, and when this has been corroborated by data received from othervehicles, the server may determine that the new data should trigger anupdate to the model.

The server may distribute the updated model (or the updated portion ofthe model) to one or more vehicles that are traveling on the roadsegment, with which the updates to the model are associated. The servermay also distribute the updated model to vehicles that are about totravel on the road segment, or vehicles whose planned trip includes theroad segment, with which the updates to the model are associated. Forexample, while an autonomous vehicle is traveling along another roadsegment before reaching the road segment with which an update isassociated, the server may distribute the updates or updated model tothe autonomous vehicle before it reaches the road segment.

In some embodiments, the remote server may collect trajectories andlandmarks from multiple clients (e.g., vehicles that travel along acommon road segment). The server may match curves using landmarks andcreate an average road model based on the trajectories collected fromthe multiple vehicles. The server may also compute a graph of roads andthe most probable path at each node or conjunction of the road segment.

The server may average landmark properties received from multiplevehicles that travelled along the common road segment, such as thedistances between one landmark to another (e.g., a previous one alongthe road segment) as measured by multiple vehicles, to determine anarc-length parameter and support localization along the path and speedcalibration for each client vehicle. The server may average the physicaldimensions of a landmark measured by multiple vehicles travelled alongthe common road segment and recognized the same landmark. The averagedphysical dimensions may be used to support distance estimation, such asthe distance from the vehicle to the landmark. The server may averagelateral positions of a landmark (e.g., position from the lane in whichvehicles are travelling in to the landmark) measured by multiplevehicles travelled along the common road segment and recognized the samelandmark. The averaged lateral portion may be used to support laneassignment. The server may average the GPS coordinates of the landmarkmeasured by multiple vehicles travelled along the same road segment andrecognized the same landmark. The averaged GPS coordinates of thelandmark may be used to support global localization or positioning ofthe landmark in the road model.

In some embodiments, the server may identify model changes, such asconstructions, detours, new signs, removal of signs, etc., based on datareceived from the vehicles. The server may continuously or periodicallyor instantaneously update the model upon receiving new data from thevehicles. The server may distribute updates to the model or the updatedmodel to vehicles for providing autonomous navigation.

In some embodiments, the server may analyze driver interventions duringthe autonomous driving. The server may analyze data received from thevehicle at the time and location where intervention occurs, and/or datareceived prior to the time the intervention occurred. The server mayidentify certain portions of the data that caused or are closely relatedto the intervention, for example, data indicating a temporary laneclosure setup, data indicating a pedestrian in the road. The server mayupdate the model based on the identified data. For example, the servermay modify one or more trajectories stored in the model.

Consistent with disclosed embodiments, the system can store informationobtained during autonomous navigation (or regular driver-controllednavigation) for use in later traversals along the same road. The systemmay share that information with other vehicles when they navigate alongthe road. Each client system may then further refine the crowd sourceddata based on observed local conditions.

FIG. 12 is a schematic illustration of a system that uses crowd sourcingdata for autonomous vehicle navigation. FIG. 12 shows a road segment1200 that includes one or more lanes. A plurality of vehicles 1205,1210, 1215, 1220, and 1225 may travel on road segment 1200 at the sametime or at different times (although shown as appearing on road segment1200 at the same time in FIG. 12). At least one of vehicles 1205-1225may be an autonomous vehicle. For simplicity of the present example, allof the vehicles 1205-1225 are presumed to be autonomous vehicles. Eachvehicle may be similar to vehicles disclosed in other embodiments (e.g.,vehicle 200), and may include components or devices included in orassociated with vehicles disclosed in other embodiments. Each vehiclemay be equipped with an image capture device or camera (e.g., imagecapture device 122 or camera 122). Each vehicle may communicate with aremote server 1230 via one or more networks (e.g., over a cellularnetwork and/or the Internet, etc.) through wireless communication paths1235, as indicated by the dashed lines. Each vehicle may transmit datato server 1230 and receive data from server 1230. For example, server1230 may collect data from multiple vehicles travelling on the roadsegment 1200 at different times, and may process the collected data togenerate an autonomous vehicle road navigation model, or an update tothe model. Server 1230 may transmit the autonomous vehicle roadnavigation model or the update to the model to the vehicles thattransmitted data to server 1230. Server 1230 may transmit the autonomousvehicle road navigation model or the update to the model to othervehicles that travel on road segment 1200 at later times.

As vehicles 1205-1225 travel on road segment 1200, navigationinformation collected (e.g., detected, sensed, or measured) by vehicles1205-1225 may be transmitted to server 1230. In some embodiments, thenavigation information may be associated with the common road segment1200. The navigation information may include a trajectory associatedwith each of the vehicles 1205-1225 as each vehicle travels over roadsegment 1200. In some embodiments, the trajectory may be reconstructedbased on data sensed by various sensors and devices provided on vehicle1205. For example, the trajectory may be reconstructed based on at leastone of accelerometer data, speed data, landmarks data, road geometry orprofile data, vehicle positioning data, and ego motion data. In someembodiments, the trajectory may be reconstructed based on data frominertial sensors, such as accelerometer, and the velocity of vehicle1205 sensed by a speed sensor. In addition, in some embodiments, thetrajectory may be determined (e.g., by a processor onboard each ofvehicles 1205-1225) based on sensed ego motion of the camera, which mayindicate three dimensional translation and/or three dimensionalrotations (or rotational motions). The ego motion of the camera (andhence the vehicle body) may be determined from analysis of one or moreimages captured by the camera.

In some embodiments, the trajectory of vehicle 1205 may be determined bya processor provided aboard vehicle 1205 and transmitted to server 1230.In other embodiments, server 1230 may receive data sensed by the varioussensors and devices provided in vehicle 1205, and determine thetrajectory based on the data received from vehicle 1205.

In some embodiments, the navigation information transmitted fromvehicles 1205-1225 to server 1230 may include data regarding the roadgeometry or profile. The geometry of road segment 1200 may include lanestructure and/or landmarks. The lane structure may include the totalnumber of lanes of road segment 1200, the type of lanes (e.g., one-waylane, two-way lane, driving lane, passing lane, etc.), markings onlanes, width of lanes, etc. In some embodiments, the navigationinformation may include a lane assignment, e.g., which lane of aplurality of lanes a vehicle is traveling in. For example, the laneassignment may be associated with a numerical value “3” indicating thatthe vehicle is traveling on the third lane from the left or right. Asanother example, the lane assignment may be associated with a text value“center lane” indicating the vehicle is traveling on the center lane.

Server 1230 may store the navigation information on a non-transitorycomputer-readable medium, such as a hard drive, a compact disc, a tape,a memory, etc. Server 1230 may generate (e.g., through a processorincluded in server 1230) at least a portion of an autonomous vehicleroad navigation model for the common road segment 1200 based on thenavigation information received from the plurality of vehicles1205-1225. Server 1230 may determine a trajectory associated with eachlane based on crowd sourced data (e.g., navigation information) receivedfrom multiple vehicles (e.g., 1205-1225) that travel on a lane of roadsegment at different times. Server 1230 may generate the autonomousvehicle road navigation model or a portion of the model (e.g., anupdated portion) based on a plurality of trajectories determined basedon the crowd sourced navigation data. Server 1230 may transmit the modelor the updated portion of the model to one or more of autonomousvehicles 1205-1225 traveling on road segment 1200 or any otherautonomous vehicles that travel on road segment at a later time forupdating an existing autonomous vehicle road navigation model providedin a navigation system of the vehicles. The autonomous vehicle roadnavigation model may be used by the autonomous vehicles in autonomouslynavigating along the common road segment 1200.

In some embodiments, the autonomous vehicle road navigation model may beincluded in a sparse map (e.g., sparse map 800 depicted in FIG. 8).Sparse map 800 may include sparse recording of data related to roadgeometry and/or landmarks along a road, which may provide sufficientinformation for guiding autonomous navigation of an autonomous vehicle,yet does not require excessive data storage. In some embodiments, theautonomous vehicle road navigation model may be stored separately fromsparse map 800, and may use map data from sparse map 800 when the modelis executed for navigation. In some embodiments, the autonomous vehicleroad navigation model may use map data included in sparse map 800 fordetermining target trajectories along road segment 1200 for guidingautonomous navigation of autonomous vehicles 1205-1225 or other vehiclesthat later travel along road segment 1200. For example, when theautonomous vehicle road navigation model is executed by a processorincluded in a navigation system of vehicle 1205, the model may cause theprocessor to compare the trajectories determined based on the navigationinformation received from vehicle 1205 with predetermined trajectoriesincluded in sparse map 800 to validate and/or correct the currenttraveling course of vehicle 1205.

In the autonomous vehicle road navigation model, the geometry of a roadfeature or target trajectory may be encoded by a curve in athree-dimensional space. In one embodiment, the curve may be a threedimensional spline including one or more connecting three dimensionalpolynomials. As one of skill in the art would understand, a spline maybe a numerical function that is piece-wise defined by a series ofpolynomials for fitting data. A spline for fitting the three dimensionalgeometry data of the road may include a linear spline (first order), aquadratic spline (second order), a cubic spline (third order), or anyother splines (other orders), or a combination thereof. The spline mayinclude one or more three dimensional polynomials of different ordersconnecting (e.g., fitting) data points of the three dimensional geometrydata of the road. In some embodiments, the autonomous vehicle roadnavigation model may include a three dimensional spline corresponding toa target trajectory along a common road segment (e.g., road segment1200) or a lane of the road segment 1200.

The autonomous vehicle road navigation model may include otherinformation, such as identification of at least one landmark along roadsegment 1200. The landmark may be visible within a field of view of acamera (e.g., camera 122) installed on each of vehicles 1205-1225. Insome embodiments, camera 122 may capture an image of a landmark. Aprocessor (e.g., processor 180, 190, or processing unit 110) provided onvehicle 1205 may process the image of the landmark to extractidentification information for the landmark. The landmark identificationinformation, rather than an actual image of the landmark, may be storedin sparse map 800. The landmark identification information may requiremuch less storage space than an actual image. Other sensors or systems(e.g., GPS system) may also provide certain identification informationof the landmark (e.g., position of landmark). The landmark may includeat least one of a traffic sign, an arrow marking, a lane marking, adashed lane marking, a traffic light, a stop line, a directional sign(e.g., a highway exit sign with an arrow indicating a direction, ahighway sign with arrows pointing to different directions or places), alandmark beacon, or a lamppost. A landmark beacon refers to a device(e.g., an RFID device) installed along a road segment that transmits orreflects a signal to a receiver installed on a vehicle, such that whenthe vehicle passes by the device, the beacon received by the vehicle andthe location of the device (e.g., determined from GPS location of thedevice) may be used as a landmark to be included in the autonomousvehicle road navigation model and/or the sparse map 800.

The identification of at least one landmark may include a position ofthe at least one landmark. The position of the landmark may bedetermined based on position measurements performed using sensor systems(e.g., Global Positioning Systems, inertial based positioning systems,landmark beacon, etc.) associated with the plurality of vehicles1205-1225. In some embodiments, the position of the landmark may bedetermined by averaging the position measurements detected, collected,or received by sensor systems on different vehicles 1205-1225 throughmultiple drives. For example, vehicles 1205-1225 may transmit positionmeasurements data to server 1230, which may average the positionmeasurements and use the averaged position measurement as the positionof the landmark. The position of the landmark may be continuouslyrefined by measurements received from vehicles in subsequent drives.

The identification of the landmark may include a size of the landmark.The processor provided on a vehicle (e.g., 1205) may estimate thephysical size of the landmark based on the analysis of the images.Server 1230 may receive multiple estimates of the physical size of thesame landmark from different vehicles over different drives. Server 1230may average the different estimates to arrive at a physical size for thelandmark, and store that landmark size in the road model. The physicalsize estimate may be used to further determine or estimate a distancefrom the vehicle to the landmark. The distance to the landmark may beestimated based on the current speed of the vehicle and a scale ofexpansion based on the position of the landmark appearing in the imagesrelative to the focus of expansion of the camera. For example, thedistance to landmark may be estimated by Z=V*dt*R/D, where V is thespeed of vehicle, R is the distance in the image from the landmark attime t1 to the focus of expansion, and D is the change in distance forthe landmark in the image from t1 to t2. dt represents the (t2−t1). Forexample, the distance to landmark may be estimated by Z=V*dt*R/D, whereV is the speed of vehicle, R is the distance in the image between thelandmark and the focus of expansion, dt is a time interval, and D is theimage displacement of the landmark along the epipolar line. Otherequations equivalent to the above equation, such as Z=V*ω/Δω, may beused for estimating the distance to the landmark. Here, V is the vehiclespeed, ω is an image length (like the object width), and Δω is thechange of that image length in a unit of time.

When the physical size of the landmark is known, the distance to thelandmark may also be determined based on the following equation:Z=f*W/ω, where f is the focal length, W is the size of the landmark(e.g., height or width), ω is the number of pixels when the landmarkleaves the image. From the above equation, a change in distance Z may becalculated using ΔZ=f*W*Δω/ω²+f*ΔW/ω, where ΔW decays to zero byaveraging, and where Δω is the number of pixels representing a boundingbox accuracy in the image. A value estimating the physical size of thelandmark may be calculated by averaging multiple observations at theserver side. The resulting error in distance estimation may be verysmall. There are two sources of error that may occur when using theformula above, namely ΔW and Δω. Their contribution to the distanceerror is given by ΔZ=f*W*Δω/ω²+f*ΔW/ω. However, ΔW decays to zero byaveraging; hence ΔZ is determined by Δω (e.g., the inaccuracy of thebounding box in the image).

For landmarks of unknown dimensions, the distance to the landmark may beestimated by tracking feature points on the landmark between successiveframes. For example, certain features appearing on a speed limit signmay be tracked between two or more image frames. Based on these trackedfeatures, a distance distribution per feature point may be generated.The distance estimate may be extracted from the distance distribution.For example, the most frequent distance appearing in the distancedistribution may be used as the distance estimate. As another example,the average of the distance distribution may be used as the distanceestimate.

FIG. 13 illustrates an example autonomous vehicle road navigation modelrepresented by a plurality of three dimensional splines 1301, 1302, and1303. The curves 1301-1303 shown in FIG. 13 are for illustration purposeonly. Each spline may include one or more three dimensional polynomialsconnecting a plurality of data points 1310. Each polynomial may be afirst order polynomial, a second order polynomial, a third orderpolynomial, or a combination of any suitable polynomials havingdifferent orders. Each data point 1310 may be associated with thenavigation information received from vehicles 1205-1225. In someembodiments, each data point 1310 may be associated with data related tolandmarks (e.g., size, location, and identification information oflandmarks) and/or road signature profiles (e.g., road geometry, roadroughness profile, road curvature profile, road width profile). In someembodiments, some data points 1310 may be associated with data relatedto landmarks, and others may be associated with data related to roadsignature profiles.

FIG. 14 illustrates a block diagram of server 1230. Server 1230 mayinclude a communication unit 1405, which may include both hardwarecomponents (e.g., communication control circuits, switches, andantenna), and software components (e.g., communication protocols,computer codes). Server 1230 may communicate with vehicles 1205-1225through communication unit 1405. For example, server 1230 may receive,through communication unit 1405, navigation information transmitted fromvehicles 1205-1225. Server 1230 may distribute, through communicationunit 1405, the autonomous vehicle road navigation model to one or moreautonomous vehicles.

Server 1230 may include one or more storage devices 1410, such as a harddrive, a compact disc, a tape, etc. Storage device 1410 may beconfigured to store data, such as navigation information received fromvehicles 1205-1225 and/or the autonomous vehicle road navigation modelthat server 1230 generates based on the navigation information. Storagedevice 1410 may be configured to store any other information, such as asparse map (e.g., sparse map 800 discussed in connection with FIG. 8).

In addition to or in place of storage device 1410, server 1230 mayinclude a memory 1415. Memory 1415 may be similar to or different frommemory 140 or 150. Memory 1415 may be a non-transitory memory, such as aflash memory, a random access memory, etc. Memory 1415 may be configuredto store data, such as computer codes or instructions executable by aprocessor (e.g., processor 1420), map data (e.g., data of sparse map800), the autonomous vehicle road navigation model, and/or navigationinformation received from vehicles 1205-1225.

Server 1230 may include a processor 1420 configured to execute computercodes or instructions stored in memory 1415 to perform variousfunctions. For example, processor 1420 may analyze the navigationinformation received from vehicles 1205-1225, and generate theautonomous vehicle road navigation model based on the analysis.Processor 1420 may control communication unit 1405 to distribute theautonomous vehicle road navigation model to one or more autonomousvehicles (e.g., one or more of vehicles 1205-1225 or any vehicle thattravels on road segment 1200 at a later time). Processor 1420 may besimilar to or different from processor 180, 190, or processing unit 110.

FIG. 15 illustrates a block diagram of memory 1415, which may storecomputer codes or instructions for performing one or more operations forprocessing vehicle navigation information for use in autonomous vehiclenavigation. As shown in FIG. 15, memory 1415 may store one or moremodules for performing the operations for processing vehicle navigationinformation. For example, memory 1415 may include a model generatingmodule 1505 and a model distributing module 1510. Processor 1420 mayexecute the instructions stored in any of modules 1505 and 1510 includedin memory 1415.

Model generating module 1505 may store instructions which, when executedby processor 1420, may generate at least a portion of an autonomousvehicle road navigation model for a common road segment (e.g., roadsegment 1200) based on navigation information received from vehicles1205-1225. For example, in generating the autonomous vehicle roadnavigation model, processor 1420 may cluster vehicle trajectories alongthe common road segment 1200 into different clusters. Processor 1420 maydetermine a target trajectory along the common road segment 1200 basedon the clustered vehicle trajectories for each of the differentclusters. Such an operation may include finding a mean or averagetrajectory of the clustered vehicle trajectories (e.g., by averagingdata representing the clustered vehicle trajectories) in each cluster.In some embodiments, the target trajectory may be associated with asingle lane of the common road segment 1200. The autonomous vehicle roadnavigation model may include a plurality of target trajectories eachassociated with a separate lane of the common road segment 1200. In someembodiments, the target trajectory may be associated with the commonroad segment 1200 instead of a single lane of the road segment 1200. Thetarget trajectory may be represented by a three dimensional spline. Insome embodiments, the spline may be defined by less than 10 kilobytesper kilometer, less than 20 kilobytes per kilometer, less than 100kilobytes per kilometer, less than 1 megabyte per kilometer, or anyother suitable storage size per kilometer.

The road model and/or sparse map may store trajectories associated witha road segment. These trajectories may be referred to as targettrajectories, which are provided to autonomous vehicles for autonomousnavigation. The target trajectories may be received from multiplevehicles, or may be generated based on actual trajectories orrecommended trajectories (actual trajectories with some modifications)received from multiple vehicles. The target trajectories included in theroad model or sparse map may be continuously updated (e.g., averaged)with new trajectories received from other vehicles.

Vehicles travelling on a road segment may collect data by varioussensors. The data may include landmarks, road signature profile, vehiclemotion (e.g., accelerometer data, speed data), vehicle position (e.g.,GPS data), and may either reconstruct the actual trajectoriesthemselves, or transmit the data to a server, which will reconstruct theactual trajectories for the vehicles. In some embodiments, the vehiclesmay transmit data relating to a trajectory (e.g., a curve in anarbitrary reference frame), landmarks data, and lane assignment alongtraveling path to server 1230. Various vehicles travelling along thesame road segment at multiple drives may have different trajectories.Server 1230 may identify routes or trajectories associated with eachlane from the trajectories received from vehicles through a clusteringprocess.

FIG. 16 illustrates a process of clustering vehicle trajectoriesassociated with vehicles 1205-1225 for determining a target trajectoryfor the common road segment (e.g., road segment 1200). The targettrajectory or a plurality of target trajectories determined from theclustering process may be included in the autonomous vehicle roadnavigation model or sparse map 800. In some embodiments, vehicles1205-1225 traveling along road segment 1200 may transmit a plurality oftrajectories 1600 to server 1230. In some embodiments, server 1230 maygenerate trajectories based on landmark, road geometry, and vehiclemotion information received from vehicles 1205-1225. To generate theautonomous vehicle road navigation model, server 1230 may clustervehicle trajectories 1600 into a plurality of clusters 1605-1630, asshown in FIG. 16.

Clustering may be performed using various criteria. In some embodiments,all drives in a cluster may be similar with respect to the absoluteheading along the road segment 1200. The absolute heading may beobtained from GPS signals received by vehicles 1205-1225. In someembodiments, the absolute heading may be obtained using dead reckoning.Dead reckoning, as one of skill in the art would understand, may be usedto determine the current position and hence heading of vehicles1205-1225 by using previously determined position, estimated speed, etc.Trajectories clustered by absolute heading may be useful for identifyingroutes along the roadways.

In some embodiments, all the drives in a cluster may be similar withrespect to the lane assignment (e.g., in the same lane before and aftera junction) along the drive on road segment 1200. Trajectories clusteredby lane assignment may be useful for identifying lanes along theroadways. In some embodiments, both criteria (e.g., absolute heading andlane assignment) may be used for clustering.

In each cluster 1605-1630, trajectories may be averaged to obtain atarget trajectory associated with the specific cluster. For example, thetrajectories from multiple drives associated with the same lane clustermay be averaged. The averaged trajectory may be a target trajectoryassociate with a specific lane. To average a cluster of trajectories,server 1230 may select a reference frame of an arbitrary trajectory C0.For all other trajectories (C1, . . . Cn), server 1230 may find a rigidtransformation that maps Ci to C0, where i=1, 2, . . . n, where n is apositive integer number, corresponding to the total number oftrajectories included in the cluster. Server 1230 may compute a meancurve or trajectory in the C0 reference frame.

In some embodiments, the landmarks may define an arc length matchingbetween different drives, which may be used for alignment oftrajectories with lanes. In some embodiments, lane marks before andafter a junction may be used for alignment of trajectories with lanes.

To assemble lanes from the trajectories, server 1230 may select areference frame of an arbitrary lane. Server 1230 may map partiallyoverlapping lanes to the selected reference frame. Server 1230 maycontinue mapping until all lanes are in the same reference frame. Lanesthat are next to each other may be aligned as if they were the samelane, and later they may be shifted laterally.

Landmarks recognized along the road segment may be mapped to the commonreference frame, first at the lane level, then at the junction level.For example, the same landmarks may be recognized multiple times bymultiple vehicles in multiple drives. The data regarding the samelandmarks received in different drives may be slightly different. Suchdata may be averaged and mapped to the same reference frame, such as theC0 reference frame. Additionally or alternatively, the variance of thedata of the same landmark received in multiple drives may be calculated.

In some embodiments, each lane of road segment 120 may be associatedwith a target trajectory and certain landmarks. The target trajectory ora plurality of such target trajectories may be included in theautonomous vehicle road navigation model, which may be used later byother autonomous vehicles travelling along the same road segment 1200.Landmarks identified by vehicles 1205-1225 while the vehicles travelalong road segment 1200 may be recorded in association with the targettrajectory. The data of the target trajectories and landmarks may becontinuously or periodically updated with new data received from othervehicles in subsequent drives.

For localization of an autonomous vehicle, the disclosed systems andmethods may use an extended Kalman filter. The location of the vehiclemay be determined based on three dimensional position data and/or threedimensional orientation data, prediction of future location ahead ofvehicle's current location by integration of ego motion. Thelocalization of vehicle may be corrected or adjusted by imageobservations of landmarks. For example, when vehicle detects a landmarkwithin an image captured by the camera, the landmark may be compared toa known landmark stored within the road model or sparse map 800. Theknown landmark may have a known location (e.g., GPS data) along a targettrajectory stored in the road model and/or sparse map 800. Based on thecurrent speed and images of the landmark, the distance from the vehicleto the landmark may be estimated. The location of the vehicle along atarget trajectory may be adjusted based on the distance to the landmarkand the landmark's known location (stored in the road model or sparsemap 800). The landmark's position/location data (e.g., mean values frommultiple drives) stored in the road model and/or sparse map 800 may bepresumed to be accurate.

In some embodiments, the disclosed system may form a closed loopsubsystem, in which estimation of the vehicle six degrees of freedomlocation (e.g., three dimensional position data plus three dimensionalorientation data) may be used for navigating (e.g., steering the wheelof) the autonomous vehicle to reach a desired point (e.g., 1.3 secondahead in the stored). In turn, data measured from the steering andactual navigation may be used to estimate the six degrees of freedomlocation.

In some embodiments, poles along a road, such as lampposts and power orcable line poles may be used as landmarks for localizing the vehicles.Other landmarks such as traffic signs, traffic lights, arrows on theroad, stop lines, as well as static features or signatures of an objectalong the road segment may also be used as landmarks for localizing thevehicle. When poles are used for localization, the x observation of thepoles (i.e., the viewing angle from the vehicle) may be used, ratherthan the y observation (i.e., the distance to the pole) since thebottoms of the poles may be occluded and sometimes they are not on theroad plane.

FIG. 17 illustrates a navigation system for a vehicle, which may be usedfor autonomous navigation. For illustration, the vehicle is referencedas vehicle 1205. The vehicle shown in FIG. 17 may be any other vehicledisclosed herein, including, for example, vehicles 1210, 1215, 1220, and1225, as well as vehicle 200 shown in other embodiments. As shown inFIG. 12, vehicle 1205 may communicate with server 1230. Vehicle 1205 mayinclude an image capture device 122 (e.g., camera 122). Vehicle 1205 mayinclude a navigation system 1700 configured for providing navigationguidance for vehicle 1205 to travel on a road (e.g., road segment 1200).Vehicle 1205 may also include other sensors, such as a speed sensor 1720and an accelerometer 1725. Speed sensor 1720 may be configured to detectthe speed of vehicle 1205. Accelerometer 1725 may be configured todetect an acceleration or deceleration of vehicle 1205. Vehicle 1205shown in FIG. 17 may be an autonomous vehicle, and the navigation system1700 may be used for providing navigation guidance for autonomousdriving. Alternatively, vehicle 1205 may also be a non-autonomous,human-controlled vehicle, and navigation system 1700 may still be usedfor providing navigation guidance.

Navigation system 1700 may include a communication unit 1705 configuredto communicate with server 1230 through communication path 1235.Navigation system 1700 may include a GPS unit 1710 configured to receiveand process GPS signals. Navigation system 1700 may include at least oneprocessor 1715 configured to process data, such as GPS signals, map datafrom sparse map 800 (which may be stored on a storage device providedonboard vehicle 1205 or received from server 1230), road geometry sensedby a road profile sensor 1730, images captured by camera 122, and/orautonomous vehicle road navigation model received from server 1230. Theroad profile sensor 1730 may include different types of devices formeasuring different types of road profile, such as road surfaceroughness, road width, road elevation, road curvature, etc. For example,the road profile sensor 1730 may include a device that measures themotion of a suspension of vehicle 1205 to derive the road roughnessprofile. In some embodiments, the road profile sensor 1730 may includeradar sensors to measure the distance from vehicle 1205 to road sides(e.g., barrier on the road sides), thereby measuring the width of theroad. In some embodiments, the road profile sensor 1730 may include adevice configured for measuring the up and down elevation of the road.In some embodiment, the road profile sensor 1730 may include a deviceconfigured to measure the road curvature. For example, a camera (e.g.,camera 122 or another camera) may be used to capture images of the roadshowing road curvatures. Vehicle 1205 may use such images to detect roadcurvatures.

The at least one processor 1715 may be programmed to receive, fromcamera 122, at least one environmental image associated with vehicle1205. The at least one processor 1715 may analyze the at least oneenvironmental image to determine navigation information related to thevehicle 1205. The navigation information may include a trajectoryrelated to the travel of vehicle 1205 along road segment 1200. The atleast one processor 1715 may determine the trajectory based on motionsof camera 122 (and hence the vehicle), such as three dimensionaltranslation and three dimensional rotational motions. In someembodiments, the at least one processor 1715 may determine thetranslation and rotational motions of camera 122 based on analysis of aplurality of images acquired by camera 122. In some embodiments, thenavigation information may include lane assignment information (e.g., inwhich lane vehicle 1205 is travelling along road segment 1200). Thenavigation information transmitted from vehicle 1205 to server 1230 maybe used by server 1230 to generate and/or update an autonomous vehicleroad navigation model, which may be transmitted back from server 1230 tovehicle 1205 for providing autonomous navigation guidance for vehicle1205.

The at least one processor 1715 may also be programmed to transmit thenavigation information from vehicle 1205 to server 1230. In someembodiments, the navigation information may be transmitted to server1230 along with road information. The road location information mayinclude at least one of the GPS signal received by the GPS unit 1710,landmark information, road geometry, lane information, etc. The at leastone processor 1715 may receive, from server 1230, the autonomous vehicleroad navigation model or a portion of the model. The autonomous vehicleroad navigation model received from server 1230 may include at least oneupdate based on the navigation information transmitted from vehicle 1205to server 1230. The portion of the model transmitted from server 1230 tovehicle 1205 may include an updated portion of the model. The at leastone processor 1715 may cause at least one navigational maneuver (e.g.,steering such as making a turn, braking, accelerating, passing anothervehicle, etc.) by vehicle 1205 based on the received autonomous vehicleroad navigation model or the updated portion of the model.

The at least one processor 1715 may be configured to communicate withvarious sensors and components included in vehicle 1205, includingcommunication unit 1705, GPS unit 1715, camera 122, speed sensor 1720,accelerometer 1725, and road profile sensor 1730. The at least oneprocessor 1715 may collect information or data from various sensors andcomponents, and transmit the information or data to server 1230 throughcommunication unit 1705. Alternatively or additionally, various sensorsor components of vehicle 1205 may also communicate with server 1230 andtransmit data or information collected by the sensors or components toserver 1230.

In some embodiments, vehicles 1205-1225 may communicate with each other,and may share navigation information with each other, such that at leastone of the vehicles 1205-1225 may generate the autonomous vehicle roadnavigation model based on information shared by other vehicles. In someembodiments, vehicles 1205-1225 may share navigation information witheach other and each vehicle may update its own the autonomous vehicleroad navigation model provided in the vehicle. In some embodiments, atleast one of the vehicles 1205-1225 (e.g., vehicle 1205) may function asa hub vehicle. The at least one processor 1715 of the hub vehicle (e.g.,vehicle 1205) may perform some or all of the functions performed byserver 1230. For example, the at least one processor 1715 of the hubvehicle may communicate with other vehicles and receive navigationinformation from other vehicles. The at least one processor 1715 of thehub vehicle may generate the autonomous vehicle road navigation model oran update to the model based on the shared information received fromother vehicles. The at least one processor 1715 of the huh vehicle maytransmit the autonomous vehicle road navigation model or the update tothe model to other vehicles for providing autonomous navigationguidance.

FIG. 18 is a flowchart showing an example process 1800 for processingvehicle navigation information for use in autonomous vehicle navigation.Process 1800 may be performed by server 1230 or processor 1715 includedin a hub vehicle. In some embodiments, process 1800 may be used foraggregating vehicle navigation information to provide an autonomousvehicle road navigation model or to update the model. Process 1800 mayinclude receiving navigation information from a plurality of vehicles(step 1805). For example, server 1230 may receive the navigationinformation from vehicles 1205-1225. The navigation information may beassociated with a common road segment (e.g., road segment 1200) alongwhich the vehicles 1205-1225 travel. Process 1800 may include storingthe navigation information associated with the common road segment (step1810). For example, server 1230 may store the navigation information instorage device 1410 and/or memory 1415. Process 1800 may includegenerating at least a portion of an autonomous vehicle road navigationmodel based on the navigation information (step 1815). For example,server 1230 may generate at least a portion of the autonomous vehicleroad navigation model for common road segment 1200 based on thenavigation information received from vehicles 1205-1225 that travel onthe common road segment 1200. Process 1800 may further includedistributing the autonomous vehicle road navigation model to one or moreautonomous vehicles (step 1820). For example, server 1230 may distributethe autonomous vehicle road navigation model or a portion (e.g., anupdate) of the model to vehicles 1205-1225, or any other vehicles latertravel on road segment 1200 for use in autonomously navigating thevehicles along road segment 1200.

Process 1800 may include additional operations or steps. For example,generating the autonomous vehicle road navigation model may includeclustering vehicle trajectories received from vehicles 1205-1225 alongroad segment 1200 into a plurality of clusters. Process 1800 may includedetermining a target trajectory along common road segment 1200 byaveraging the clustered vehicle trajectories in each cluster. Process1800 may also include associating the target trajectory with a singlelane of common road segment 1200. Process 1800 may include determining athree dimensional spline to represent the target trajectory in theautonomous vehicle road navigation model.

FIG. 19 is a flowchart showing an example process 1900 performed by anavigation system of a vehicle. Process 1900 may be performed byprocessor 1715 included in navigation system 1700. Process 1900 mayinclude receiving, from a camera, at least one environmental imageassociated with the vehicle (step 1905). For example, processor 1715 mayreceive, from camera 122, at least one environmental image associatedwith vehicle 1205. Camera 122 may capture one or more images of theenvironment surrounding vehicle 1205 as vehicle 1205 travels along roadsegment 1200. Process 1900 may include analyzing the at least oneenvironmental image to determine navigation information related to thevehicle (step 1910). For example, processor 1715 may analyze theenvironmental images received from camera 122 to determine navigationinformation, such as a trajectory of travel along road segment 1200.Processor 1715 may determine the trajectory of travel of vehicle 1205based on camera ego motions (e.g., three dimensional translation and/orthree dimensional rotational motions) sensed by, e.g., the analysis ofthe images.

Process 1900 may include transmitting the navigation information fromthe vehicle to a server (step 1915). In some embodiments, the navigationinformation may be transmitted along with road information from thevehicle to server 1230. For example, processor 1715 may transmit, viacommunication unit 1705, the navigation information along with roadinformation, such as the lane assignment, road geometry, from vehicle1205 to server 1230. Process 1900 may include receiving from the serveran autonomous vehicle road navigation model or a portion of the model(step 1920). For example, processor 1715 may receive the autonomousvehicle road navigation model or a portion of the model from server1230. The model or the portion of the model may include at least oneupdate to the model based on the navigation information transmitted fromvehicle 1205. Processor 1715 may update an existing model provided innavigation system 1700 of vehicle 1205. Process 1900 may include causingat least one navigational maneuver by the vehicle based on theautonomous vehicle road navigation model (step 1925). For example,processor 1715 may cause vehicle 1205 to steer, make a turn, changelanes, accelerate, brake, stop, etc. Processor 1715 may send signals toat least one of throttling system 220, braking system 230, and steeringsystem 240 to cause vehicle 1205 to perform the navigational maneuver.

Process 1900 may include other operations or steps performed byprocessor 1715. For example, the navigation information may include atarget trajectory for vehicles to travel along a road segment, andprocess 1900 may include clustering, by processor 1715, vehicletrajectories related to multiple vehicles travelling on the road segmentand determining the target trajectory based on the clustered vehicletrajectories. Clustering vehicle trajectories may include clustering, byprocessor 1715, the multiple trajectories related to the vehiclestravelling on the road segment into a plurality of clusters based on atleast one of the absolute heading of vehicles or lane assignment of thevehicles. Generating the target trajectory may include averaging, byprocessor 1715, the clustered trajectories. Other processes or stepsperformed by server 1230, as described above, may also be included inprocess 1900.

The disclosed systems and methods may include other features. Forexample, the disclosed systems may use local coordinates, rather thanglobal coordinates. For autonomous driving, some systems may presentdata in world coordinates. For example, longitude and latitudecoordinates on the earth surface may be used. In order to use the mapfor steering, the host vehicle must know its position and orientationrelative to the map. It seems natural to use a GPS device on board, inorder to position the vehicle on the map and in order to find therotation transformation between the body reference frame and the worldreference frame (say, North, East and Down). Once the body referenceframe is aligned with the map reference frame, then the desired routemay be expressed in the body reference frame and the steering commandsmay be computed or generated.

However, one possible issue with this strategy is that current GPStechnology does not usually provide the body location and pose withsufficient accuracy and availability. To overcome this problem, it hasbeen proposed to use landmarks whose world coordinates are known. Theidea is to construct very detailed maps (called High Definition or HDmaps), that contain landmarks of different kinds. The assumption is thatthe vehicle is equipped with a sensor that can detect and locate thelandmarks in its own reference frame. Once the relative position betweenthe vehicle and the landmarks is found, the landmarks' world coordinatesare taken from the HD map, and the vehicle can use them to compute itsown location and pose.

This method is still using the global world coordinate system as amediator that establishes the alignment between the map and the bodyreference frames. Namely, the landmarks are used in order to compensatefor the limitations of the GPS device onboard the vehicles. Thelandmarks, together with an HD map, may enable to compute the precisevehicle pose in global coordinates, and hence the map-body alignmentproblem is solved.

In the disclosed systems and methods, instead of using one global map ofthe world, many map pieces or local maps may be used for autonomousnavigation. Each piece of a map or each local map may define its owncoordinate frame. These coordinate frames may be arbitrary. Thevehicle's coordinates in the local maps may not need to indicate wherethe vehicle is located on the surface of earth. Moreover, the local mapsmay not be required to be accurate over large scales, meaning there maybe no rigid transformation that can embed a local map in the globalworld coordinate system.

There are two main processes associated with this representation of theworld, one relates to the generation of the maps and the other relatesto using them. With respect to maps generation, this type ofrepresentation may be created and maintained by crowd sourcing. Theremay be no need to apply sophisticated survey equipment, because the useof HD maps is limited, and hence crowd sourcing becomes feasible. Withrespect to usage, an efficient method to align the local map with thebody reference frame without going through a standard world coordinatesystem may be employed. Hence there may be no need, at least in mostscenarios and circumstances, to have a precise estimation of the vehiclelocation and pose in global coordinates. The memory footprint of thelocal maps may be kept very small.

The principle underlying the maps generation is the integration of egomotion. The vehicles sense the motion of the camera in space (3Dtranslation and 3D rotation). The vehicles or the server may reconstructthe trajectory of the vehicle by integration of ego motion over time,and this integrated path may be used as a model for the road geometry.This process may be combined with sensing of close range lane marks, andthen the reconstructed route may reflect the path that a vehicle shouldfollow, and not the particular path that it did follow. In other words,the reconstructed route or trajectory may be modified based on thesensed data relating to close range lane marks, and the modifiedreconstructed trajectory may be used as a recommended trajectory ortarget trajectory, which may be saved in the road model or sparse mapfor use by other vehicles navigating the same road segment.

In some embodiments, the map coordinate system may be arbitrary. Acamera reference frame may be selected at an arbitrary time, and used asthe map origin. The integrated trajectory of the camera may be expressedin the coordinate system of that particular chosen frame. The value ofthe route coordinates in the map may not directly represent a locationon earth.

The integrated path may accumulate errors. This may be due to the factthat the sensing of the ego motion may not be absolutely accurate. Theresult of the accumulated error is that the local map may diverge, andthe local map may not be regarded as a local copy of the global map. Thelarger the size of the local map piece, the larger the deviation fromthe “true” geometry on earth.

The arbitrariness and the divergence of the local maps may not be adesign principle but rather may be a consequence. These properties maybe a consequence of the integration method, which may be applied inorder to construct the maps in a crowd sourcing manner (by vehiclestraveling along the roads). However, vehicles may successfully use thelocal maps for steering.

The proposed map may diverge over long distances. Since the map is usedto plan a trajectory in the immediate vicinity of the vehicle, theeffect of the divergence may be acceptable. At any time instance, thesystem (e.g., server 1230 or vehicle 1205) may repeat the alignmentprocedure, and use the map to predict the road location (in the cameracoordinate frame) some 1.3 seconds ahead (or any other seconds, such as1.5 seconds, 1.0 second, 1.8 seconds, etc.). As long as the accumulatederror over that distance is small enough, then the steering commandprovided for autonomous driving may be used.

In some embodiments, a local map may focus on a local area, and may notcover a too large area. This means that a vehicle that is using a localmap for steering in autonomous driving, may arrive at some point to theend of the map and may have to switch to another local piece of map. Theswitching may be enabled by the local maps overlapping each other. Oncethe vehicle enters the area that is common to both maps, the system(e.g., server 1230 or vehicle 1205) may continue to generate steeringcommands based on a first local map (the map that is being used), but atthe same time the system may localize the vehicle on the other map (orsecond local map) that overlaps with the first local map. In otherwords, the system may simultaneously align the present coordinate frameof the camera both with the coordinate frame of the first map and withthe coordinate frame of the second map. When the new alignment isestablished, the system may switch to the other map and plan the vehicletrajectory there.

The disclosed systems may include additional features, one of which isrelated to the way the system aligns the coordinate frames of thevehicle and the map. As explained above that landmarks may be used foralignment, assuming the vehicle may measure its relative position tothem. This is useful in autonomous driving, but sometimes it may resultin a demand for a large number of landmarks and hence a large memoryfootprint. The disclosed systems may therefore use an alignmentprocedure that addresses this problem. In the alignment procedure, thesystem may compute a 1D estimator for the location of the vehicle alongthe road, using sparse landmarks and integration of ego speed. Thesystem may use the shape of the trajectory itself to compute therotation part of the alignment, using a tail alignment method discussedin details below in other sections. The idea is that the vehiclereconstructs its own trajectory while driving the “tail” and computes arotation around its assumed position along the road, in order to alignthe tail with the map.

In the disclosed systems and methods, a GPS device may still be used.Global coordinates may be used for indexing the database that stores thetrajectories and/or landmarks. The relevant piece of local map and therelevant landmarks in the vicinity of the vehicles may be stored inmemory and retrieved from the memory using global GPS coordinates.However, in some embodiments, the global coordinates may not be used forpath planning, and may not be accurate. In one example, the usage ofglobal coordinates may be limited for indexing of the information.

In situations where “tail alignment” cannot function well, the systemmay compute the vehicle's pose using a larger number of landmarks. Thismay be a rare case, and hence the impact on the memory footprint may bemoderate. Road intersections are examples of such situations.

The disclosed systems and methods may use semantic landmarks (e.g.,traffic signs), since they can be reliably detected from the scene andmatched with the landmarks stored in the road model or sparse map. Insome cases the disclosed systems may use non-semantic landmarks (e.g.,general purpose signs) as well, and in such cases the non-semanticlandmarks may be attached to an appearance signature, as discussedabove. The system may use a learning method for the generation ofsignatures that follows the “same or not-same” recognition paradigm.

For example, given many drives with GPS coordinates along them, thedisclosed systems may produce the underlying road structure junctionsand road segments. The roads are assumed to be far enough from eachother to be able to differentiate them using the GPS. Only a coarsegrained map may be needed. To generate the underlying road structuregraph, the space may be divided into a lattice of a given resolution(e.g., 50 m by 50 m). Every drive may be seen as an ordered list oflattice sites. The system may color every lattice site belonging to adrive to produce an image of the merged drives. The colored latticepoints may be represented as nodes on the merged drives. The drivespassing from one node to another may be represented as links. The systemmay fill small holes in the image, to avoid differentiating lanes andcorrect for GPS errors. The system may use a suitable thinning algorithm(e.g., an algorithm named “Zhang-Suen” thinning algorithm) to obtain theskeleton of the image. This skeleton may represent the underlying roadstructure, and junctions may be found using a mask (e.g., a pointconnected to at least three others). After the junctions are found, thesegments may be the skeleton parts that connect them. To match thedrives hack to the skeleton, the system may use a Hidden Markov Model.Every GPS point may be associated with a lattice site with a probabilityinverse to its distance from that site. Use a suitable algorithm (e.g.,an algorithm named the “Viterbi” algorithm) to match GPS points tolattice sites, while not allowing consecutive GPS points to match to nonneighboring lattice sites.

A plurality of methods may be used for mapping the drives back to themap. For example, a first solution may include keeping track during thethinning process. A second solution may use proximity matching. A thirdsolution may use hidden Markov model. The hidden Markov model assumes anunderlying hidden state for every observation, and assigns probabilitiesfor a given observation given the state, and for a state given theprevious state. A Viterbi algorithm may be used to find the mostprobable states given a list of observations.

The disclosed systems and methods may include additional features. Forexample, the disclosed systems and methods may detect highwayentrances/exits. Multiple drives in the same area may be merged usingGPS data to the same coordinate system. The system may use visualfeature points for mapping and localization.

In some embodiments, generic visual features may be used as landmarksfor the purpose of registering the position and orientation of a movingvehicle, in one drive (localization phase), relative to a map generatedby vehicles traversing the same stretch of road in previous drives(mapping phase). These vehicles may be equipped with calibrated camerasimaging the vehicle surroundings and GPS receivers. The vehicles maycommunicate with a central server (e.g., server 1230) that maintains anup-to-date map including these visual landmarks connected to othersignificant geometric and semantic information (e.g. lane structure,type and position of road signs, type and position of road marks, shapeof nearby drivable ground area delineated by the position of physicalobstacles, shape of previously driven vehicle path when controlled byhuman driver, etc.). The total amount of data that may be communicatedbetween the central server and vehicles per length of road is small,both in the mapping and localization phases.

In the mapping phase, the disclosed systems (e.g., vehicles or server)may detect feature points (FPs) and compute their descriptors (e.g.using the FAST/BRISK/ORB detectors and descriptors or adetector/descriptor pair that was trained using the database discussedbelow). The system may track FPs between frames in which they appearusing their motion in the image plane and by matching their descriptorsusing e.g. Euclidean or Hamming distance in descriptor space. The systemmay use tracked FPs to estimate camera motion and world positions ofobjects on which FPs were detected and tracked. The system may classifyFPs as ones that will likely be detected in future drives (e.g. FPsdetected on momentarily moving objects, parked cars and shadow texturewill likely not reappear in future drives). This reproducibilityclassification (RC) may be a function of both the intensities in aregion of the image pyramid surrounding the detected FP, the motion ofthe tracked FP in the image plane, the extent of viewpoints in which itwas successfully tracked/detected and its relative 3D position. In someembodiments, the vehicles may send representative FP descriptors(computed from a set of observations), estimated 3D position relative tovehicle and momentary vehicle GPS coordinates to server 1230.

During the mapping phase, when communication bandwidth between themapping vehicles and central server is limited, the vehicles may sendFPs to the server at a high frequency when the presence of FPs or othersemantic landmarks in the map (such as road signs and lane structure) islimited and insufficient for the purpose of localization. Althoughvehicles in the mapping phase may send FPs at a low spatial frequencythese may be agglomerated in the server. Detection of reoccurring FPsmay also be performed by the server and the server may store the set ofreoccurring FPs. Visual appearance of landmarks may at least in somecases be sensitive to the time of day or the season in which they werecaptured. To increase reproducibility probability of FPs, these may bebinned by the server into time-of-day and season bins.

The vehicles may send the server other semantic and geometricinformation in the nearby FP coordinate system (lane shape, structure ofroad plane, 3D position of obstacles, free space in mapping clipmomentary coordinate system, path driven by human driver in a setupdrive to a parking location).

In a localization phase, the server may send a map containing landmarksin the form of FP positions and descriptors to vehicles. Feature points(FPs) may be detected and tracked in near real time within a set ofcurrent consecutive frames. Tracked FPs may be used to estimate cameramotion and world positions of FPs. Currently detected FP descriptors maybe searched to match a list of map FPs having GPS coordinates within anestimated finite GPS uncertainty radius from the momentary GPS reading.Matching may be done by searching all pairs of current and mapping FPsthat minimize an Euclidean or Hamming distance in descriptor space.Using the FP matches and their current and map positions, rotation andtranslation between the momentary vehicle position and the local mapcoordinate system may be registered.

The disclosed systems and methods may include a method for training areproducibility classifier. Training may be performed in one of thefollowing schemes in order of growing labeling cost and resultingclassifier accuracy.

In the first scheme, a database including a large number of clipsrecorded by vehicle cameras with matching momentary vehicle GPS positionmay be collected. This database may include a representative sample ofdrives (with respect to various properties: e.g., time of day, season,weather condition, type of roadway). Feature points (FPs) extracted fromframes of different drives at a similar GPS position and heading may bepotentially matched within a GPS uncertainty radius. Unmatched FPs maybe labeled unreproducible and those matched may be labeled reproducible.A classifier may then be trained to predict the reproducibility label ofan FP given its appearance in the image pyramid, its momentary positionrelative to the vehicle and the extent of viewpoints positions in whichit was successfully tracked.

In the second scheme, FP pairs extracted from the clip databasedescribed in the first scheme may also be labeled by a human responsiblefor annotating FP matches between clips.

In a third scheme, a database augmenting that of the first scheme withprecise vehicle position, vehicle orientation and image pixel depthusing Light Detection And Ranging (LIDAR) measurements may be used toaccurately match world positions in different drives. Feature pointdescriptors may then be computed at the image region corresponding tothese world points at different viewpoints and drive times. Theclassifier may then be trained to predict the average distance indescriptor space a descriptor is located from its matched descriptors.In this case reproducibility may be measured by likely having a lowdescriptor distance.

Uploading Recommended, Not Actual Trajectories

Consistent with disclosed embodiments, the system may generate anautonomous vehicle road navigation model based on the observedtrajectories of vehicles traversing a common road segment (e.g., whichmay correspond to the trajectory information forwarded to a server by avehicle). The observed trajectories, however, may not correspond toactual trajectories taken by vehicles traversing a road segment. Rather,in certain situations, the trajectories uploaded to the server may bemodified with respect to actual reconstructed trajectories determined bythe vehicles. For example, a vehicle system, while reconstructing atrajectory actually taken, may use sensor information (e.g., analysis ofimages provided by a camera) to determine that its own trajectory maynot be the preferred trajectory for a road segment. For example, thevehicle may determine based on image data from onboard cameras that itis not driving in a center of a lane or that it crossed over a laneboundary for a determined period of time. In such cases, among others, arefinement to the vehicle's reconstructed trajectory (the actual pathtraversed) may be made based on information derived from the sensoroutput. The refined trajectory, not the actual trajectory, may then beuploaded to the server for potential use in building or updating sparsedata map 800.

Referring to FIGS. 12 and 17, vehicle 1205 may communicate with server1230. Vehicle 1205 may be an autonomous vehicle or a traditional,primarily human-controlled vehicle. Vehicle 1205 may collect (or detect,sense, measure) data regarding road segment 1200 as vehicle 1205 travelsalong road segment 1200. The collected data may include navigationinformation, such as road geometry, recognized landmark including signs,road markings, etc. Vehicle 1205 may transmit the collected data toserver 1230. Server 1230 may generate and/or update an autonomousvehicle road navigation model based on the data received from vehicle1205. The autonomous vehicle road navigation model may include aplurality of target trajectories representing preferred paths of travelalong particular road segments.

As shown in FIG. 17, vehicle 1205 may include navigation system 1700.Navigation system may include a storage device (e.g., a hard drive, amemory) configured for storing the autonomous vehicle road navigationmodel and/or map data (e.g., map data of sparse map 800). It should benoted that the storage device may store a local copy of the entire roadmodel from sparse data map 800. Alternately, the storage device maystore only portions of sparse data maps (e.g., local maps) provided tothe navigating vehicle as needed. In such embodiments, the local mapsmay be stored only temporarily in the storage device and may be purgedfrom the storage device upon receipt of one or more newly received localmaps or after a vehicle is determined to have exited a particularnavigational area or zone. Navigation system 1700 may include at leastone processor 1715.

Navigation system 1700 may include one or more sensors, such as camera122, GPS unit 1710, road profile sensor 1730, speed sensor 1720, andaccelerometer 1725. Vehicle 1205 may include other sensors, such asradar sensors. The sensors included in vehicle 1205 may collect datarelated to road segment 1200 as vehicle 1205 travels along road segment1200.

The processor 1715 may be configured to receive, from the one or moresensors, outputs indicative of a motion of vehicle 1205. For example,accelerometer 1725 may output signals indicating three dimensionaltranslation and/or three dimensional rotational motions of camera 122.Speed sensor may output a speed of vehicle 1205. Road profile sensor1730 may output signals indicating road roughness, road width, roadelevation, road curvature, which may be used to determine the motion ortrajectory of the vehicle 1205.

Processor 1715 may determine an actual trajectory of vehicle 1205 basedon the outputs from the one or more sensors. For example, based onanalysis of images output from camera 122, processor 1715 may identifylandmarks along road segment 1200. Landmarks may include traffic signs(e.g., speed limit signs), directional signs (e.g., highway directionalsigns pointing to different routes or places), and general signs (e.g.,a rectangular business sign that is associated with a unique signature,such as a color pattern). The identified landmark may be compared withthe landmark stored in sparse map 800. When a match is found, thelocation of the landmark stored in sparse map 800 may be used as thelocation of the identified landmark. The location of the identifiedlandmark may be used for determining the location of the vehicle 1205along a target trajectory. In some embodiments, processor 1715 may alsodetermine the location of vehicle 1205 based on (IPS signals output byGPS unit 1710.

Processor 1715 may determine the vehicle motion based on output from theaccelerometer 1725, the camera 122, and/or the speed sensor 1720. Forexample, speed sensor 1720 may output a current speed of vehicle 1205 toprocessor 1715. Accelerometer 1725 may output a signal indicating threedimensional translation and/or rotation of vehicle 1205 to processor1715. The camera 122 may output a plurality of images of the surroundingof vehicle 1205 to processor 1715. Based on the outputs from theplurality of sensors and devices, processor 1715 may determine an actualtrajectory of vehicle 1205. The actual trajectory reflects the actualpath vehicle 1205 has taken or is taking, including, e.g., which lanealong road segment 1200 vehicle 1205 has travelled in or is travellingin, and what different road segments vehicle 1205 have travelled along.

Processor 1715 may receive, from camera 122, at least one environmentalimage associated with vehicle 1205. For example, camera 122 may be afront-facing camera, which may capture an image of the environment infront of vehicle 1205. Camera 122 may be facing other directions, suchas the sides of vehicle 1205 or the rear of vehicle 1205. Vehicle 1205may include a plurality of cameras facing different directions.Processor 1715 may analyze the at least one environmental image todetermine information associated with at least one navigationalconstraint. The navigational constraint may include at least one of abarrier (e.g., a lane separating barrier), an object (e.g., apedestrian, a lamppost, a traffic light post), a lane marking (e.g., asolid yellow lane marking), a sign (e.g., a traffic sign, a directionalsign, a general sign), or another vehicle (e.g., a leading vehicle, afollowing vehicle, a vehicle that is traveling on the side of vehicle1205).

Processor 1715 may also determine a target trajectory for transmittingto server 1230. The target trajectory may be the same as the actualtrajectory determined by processor 1715 based on the sensor outputs. Insome embodiments, the target trajectory may be different from the actualtrajectory determined based on the sensor outputs. The target trajectorymay include one or more modifications to the actual trajectory based onthe determined information associated with the at least one navigationalconstraint.

For example, the environmental image captured by camera 122 may includea barrier, such as a temporary lane shifting barrier 100 meters ahead ofvehicle 1250 that changes the lanes (e.g., when lanes are temporarilyshifted due to an accident ahead). Processor 1715 may detect thetemporary lane shifting barrier from the image, and take a lanedifferent from a lane corresponding to the target trajectory stored inthe road model or sparse map in compliance to the temporary lane shift.The actual trajectory of vehicle may reflect this change of lanes.However, the lane shifting is temporary and may be cleared in the next10, 15, or 30 minutes. Vehicle 1205 may thus modify the actualtrajectory (i.e., the shift of lanes) vehicle 1205 has taken to reflectthat a target trajectory should be different from the actual trajectoryvehicle 1205 has taken. For example, the system may recognize that thepath traveled differs from a preferred trajectory for the road segment.Thus, the system may adjust a reconstructed trajectory prior touploading the trajectory information to the servers. In otherembodiments, the actual reconstructed trajectory information may beuploaded, by one or more recommended trajectory refinements (e.g., asize and direction of a translation to be made to at least a portion ofthe reconstructed trajectory) may also be uploaded. In some embodiments,processor 1715 may transmit a modified actual trajectory to server 1230.Server 1230 may generate or update a target trajectory based on thereceived information and may transmit the target trajectory to otherautonomous vehicles that later travel on the same road segment.

As another example, the environmental image may include an object, suchas a pedestrian suddenly appearing in road segment 1200. Processor 1715may detect the pedestrian, and vehicle 1205 may change lanes to avoid acollision with the pedestrian. The actual trajectory vehicle 1205reconstructed based on sensed data may include the change of lanes.However, the pedestrian may soon leave the roadway. So, vehicle 1205 maymodify the actual trajectory (or determine a recommended modification)to reflect that the target trajectory should be different from theactual trajectory taken (as the appearance of the pedestrian is atemporary condition that should not be accounted for in the targettrajectory determination. In some embodiments, the vehicle may transmitto the server data indicating a temporary deviation from thepredetermined trajectory, when the actual trajectory is modified. Thedata may indicate a cause of the deviation, or the server may analyzethe data to determine a cause of the deviation. Knowing the cause of thedeviation may be useful. For example, when the deviation is due to thedriver noticing an accident that has recently occurred and, in responsesteering the wheel to avoid collision, the server may plan a moremoderate adjustment to the model or a specific trajectory associatedwith the road segment based on the cause of deviation. As anotherexample, when the cause of deviation is a pedestrian crossing the road,the server may determine that there is no need to change the trajectoryin the future.

As another example, the environmental image may include a lane markingindicating that vehicle 1205 is driving slightly outside of a lane,perhaps under the control of a human driver. Processor 1715 may detectthe lane marking from the captured images and may modify the actualtrajectory of vehicle 1205 to account for the departure from the lane.For example, a translation may be applied to the reconstructedtrajectory so that it falls within the center of an observed lane.

FIG. 20 shows an example memory 2000. Memory 2000 may include variousmodules, which when executed by a processor, may cause the processor toperform the disclosed methods. For example, memory 2000 may include anactual trajectory determination module 2005. Actual trajectorydetermination module 2005, when executed by a processor (e.g., processor1715 or other processors), may cause the processor to determine anactual trajectory of a vehicle based on data output or received from oneor more sensors included in the vehicle. For example, the processor mayreconstruct the actual trajectory based on signals received from one ormore of accelerometer 1725, camera 122, and/or speed sensor 1720. Insome embodiments, the processor may determine the actual trajectorybased on the outputs received from the sensors indicative of a motion ofthe vehicle.

Memory 2000 may also include a target trajectory determination module2010. Target trajectory determination module 2010, when executed by theprocessor, may cause the processor to determine a target trajectorybased on the actual trajectory. For example, based on data received fromthe sensor, the processor may determine that one or more modificationsneed to be made to the actual trajectory. The modified actual trajectorymay be used as the target trajectory for transmitting to a server (e.g.,server 1230). The target trajectory may represent a better trajectorythan the actual trajectory for other autonomous vehicles to follow whenthe other autonomous vehicles travel on the same road segment at a latertime. In some embodiments, the processor may determine a targettrajectory that includes the actual trajectory and one or moremodifications based on information associated with navigationalconstraints.

Memory 2000 may also include an image analysis module 2015. Imageanalysis module, when executed by the processor, may cause the processorto analyze one or more images captured by a camera (e.g., camera 122)using various image analysis algorithms. For example, the processor mayanalyze an image of the environment to identify a landmark, at least onenavigational constraint, or to calculate a distance from the vehicle tothe landmark, etc.

FIG. 21 is a flowchart illustrating an example process for uploadingrecommended trajectory to a server. Process 2100 may be performed by aprocessor included in a navigation system of a vehicle, such asprocessor 1715 included in navigation system 1700 of autonomous vehicle1205. Process 2100 may include receiving, from one or more sensors,outputs indicative of a motion of a vehicle (step 2105). For example,processor 1715 may receive outputs from inertial sensors, such asaccelerometer 1725 indicating the three dimensional translation and/orthree dimensional rotational motions of vehicle 1205. Process 2100 mayinclude determining an actual trajectory of the vehicle based on theoutputs from the one or more sensors (step 2110). For example, processor1715 may analyze images from camera 122, speed from speed sensor 1720,position information from GPS unit 1710, motion data from accelerometer1725, to determine an actual trajectory. Process 2100 may includereceiving, from the camera, at least one environmental image associatedwith the vehicle (step 2115). For example, processor 1715 may receive atleast one environmental image associated with vehicle 1205 from camera122. Camera 122 may be a front-facing camera, which may capture an imageof an environment in front of vehicle 1205. Process 2100 may includeanalyzing the at least one environmental image to determine informationassociated with at least one navigation constraint (step 2120). Forexample, processor 1715 may analyze the environmental images from camera122 to detect at least one of a barrier, an object, a lane marking, asign, or another vehicle in the images. Process 2100 may also includedetermining a target trajectory, including the actual trajectory and oneor more modifications to the actual trajectory based on the determinedinformation associated with the navigational constraint (step 2125). Forexample, based on at least one of the barrier, object, lane marking,sign, or another vehicle detected from the environmental images,processor 1715 may modify the actual trajectory, e.g., to include a laneor a road other than the lane or road vehicle 1205 is travelling in. Themodified actual trajectory may be used as the target trajectory. Thetarget trajectory may reflect a safer or better trajectory than theactual trajectory vehicle 1205 is taking. Process 2100 may furtherinclude transmitting the target trajectory to a server (step 2130). Forexample, processor 1715 may transmit the target trajectory from vehicle1205 to server 1230. Server 1230 may transmit the target trajectoryreceived from vehicle 1205 to other vehicles (which may be autonomousvehicles or traditional, human-operated vehicles). Other vehicles maychange their lanes or paths based on the target trajectory. In someembodiments, process 2100 may include overriding a change in trajectorythat is suggested by the server. For example, when the vehicle isapproaching a lane split, and the server determines to change thecurrent lane to a lane that has been temporarily closed or marked forother traffic, processor 1715 may override the determination by theserver based on the detection (e.g., from images captured by the cameraonboard the vehicle) of the temporary closure.

Process 2100 may include other operations or steps. For example,processor 1715 may receive target trajectories from server 1230. Thetarget trajectories may be transmitted to server 1230 from othervehicles travelling ahead of vehicle 1205 on the same road segment 1200.Processor 1715 may update an autonomous vehicle road navigation modelprovided in navigation system 1700 with the target trajectories receivedfrom server 1230, and cause vehicle 1205 to make a navigationalmaneuver, such as changing a lane.

Landmark Identification

Consistent with disclosed embodiments, the system may identify landmarksfor use in an autonomous vehicle road navigation model. Thisidentification may include a determination of a landmark type, physicalsize, and location of the identified landmark, among othercharacteristics.

FIG. 22 illustrates an example environment including a system foridentifying a landmark for use in autonomous vehicle navigation. In thisexample, FIG. 22 shows a road segment 2200. Vehicles 2201 and 2202 maybe traveling along road segment 2200. Along the road segment 2200, theremay be one or more signs or objects (e.g., 2205 and 2206), which may beidentified as landmarks. Landmarks may be stored in an autonomousvehicle road navigation model or a sparse map (e.g., sparse map 800).Actual images of the landmarks need not be saved in the model or sparsemap. Rather, as previously discussed, a small amount of data thatcharacterizes the landmark type, location, physical size, and, incertain cases, a condensed image signature may be stored in the model orsparse map, thereby reducing the storage space required for storing themodel or sparse map and/or transmitting some or all of the sparse map toautonomous vehicles. In addition, not every landmark appearing along aroad segment is stored. The model or sparse map may have sparserecording of recognized landmarks, which may be spaced apart from eachother along a road segment by at least 50 meters, 100 meters, 500meters, 1 kilometer, 2 kilometers, etc. Sparse recording of thelandmarks also reduce the storage space required for storing datarelating to the landmarks. Landmarks stored in the model and/or sparsemap may be used for autonomous vehicle navigation along road segment2200. For example, recognized landmarks included in sparse data map 800may be used for locating vehicles 2201 and 2202 (e.g., determininglocations of vehicles 2201 and 2202 along a target trajectory stored inthe model or sparse map).

Vehicles 2201 and 2202 may be autonomous vehicles, and may be similar tovehicles disclosed in other embodiments. For example, vehicles 2201 and2202 may include components and devices included in vehicle 200, such asat least one image capture device (e.g., image capture device or camera122). Vehicles 2201 and 2202 may each include at least one processor2210, which may be similar to processor 180, 190, or processing unit110. Each of vehicles 2201 and 2202 may include a communication unit2215, which may communicate with a server 2230 via one or more networks(e.g., over a cellular network and/or the Internet, etc.).

Server 2230 may include both hardware components (e.g., circuits,switches, network cards) and software components (e.g., communicationprotocols, computer-readable instructions or codes). For example, server2230 may include a communication unit 2231 configured to communicatewith communication units 2215 of vehicles 2201 and 2202. Server 2230 mayinclude at least one processor 2232 configured to process data, such asthe autonomous vehicle road navigation model, the sparse map (e.g.,sparse map 800), and/or navigation information received from vehicles2201 and 2202. The navigation information may include any informationreceived from vehicles 2201 and 2202, such as images of landmarks,landmark identifiers, Global Positioning System signals, ego motiondata, speed, acceleration, road geometry (e.g., road profile, lanestructure, elevation of road segment 2200), etc. Server 2230 may includea storage device 2233, which may be a hard drive, a compact disc, amemory, or other non-transitory computer readable media.

Vehicles 2201 and 2202 may capture at least one image, via camera 122,of an environment of vehicles as the vehicles travel along road segment2200. The image of the environment may include an image of signs orlandmarks 2205 and 2206. In some embodiments, at least one identifierassociated with landmarks 2205 and 2206 may be determined by vehicles2201 and 2202, and the identifier may be transmitted to server 2230 fromthe vehicles. In some embodiments, at least one identifier associatedwith landmarks 2205 and 2206 may be determined by server 2230 based onimages of the landmarks 2205 and 2206 captured by cameras 122 andtransmitted to server 2230.

For example, camera 122 installed on a host vehicle (e.g., vehicle 2201hosting camera 122) may acquire at least one image representative of anenvironment of vehicle 2201 (e.g., in front of vehicle 2201). Processor2215 included in vehicle 2201 may analyze the at least one image toidentify a landmark (e.g., landmark 2206) in the environment of the hostvehicle. Processor 2215 may also analyze the at least one image todetermine the at least one identifier associated with the landmark.

In some embodiments, processor 2215 may then transmit the at least oneidentifier to server 2230. Server 2230 (e.g., through processor 2232 andcommunication unit 2231) may receive the at least one identifierassociated with the landmark. Processor 2232 may associate the landmarkwith the corresponding road segment 2200. Processor 2232 may update theautonomous vehicle road navigation model relative to the correspondingroad segment 2200 to include the at least one identifier associated withthe landmark. Processor 2232 may distribute the updated autonomousvehicle road navigation model to a plurality of autonomous vehicles,such as vehicles 2201 and 2202, and other vehicles that travel alongroad segment 2200 at later times.

The at least one identifier associated with the landmark (e.g., landmark2205 or 2206) may include a position of the landmark. The position maybe determined based on the signals provided by various sensors ordevices installed on vehicles 2201 and 2202 (e.g., UPS signals, vehiclemotion signals). The identifier may include a shape of the landmark. Forexample, the identifier may include data indicating a rectangular shapeof landmark 2205 or a triangular shape of landmark 2206. The identifiermay include a size of the landmark. For example, the identifier mayinclude data indicating a width and/or height of the rectangular sign2205 and/or the triangular sign 2206. The identifier may include adistance of the landmark relative to another landmark. For example, theidentifier associated with landmark 2206 may include a distance d fromlandmark 2206 to landmark 2205. The distance d is shown as a distancebetween landmarks 2205 and 2206 along road segment 2200. Other distancesmay also be used, such as the direct distance between landmarks 2205 and2206 crossing the road segment 2200. In some embodiments, the distancemay refer to a distance from the recognized landmark (e.g., 2206) to apreviously recognized landmark (e.g., a landmark that is recognized atleast 50 meters, 100 meters, 500 meters, 1 kilometer, 2 kilometers awayback along road segment 2200).

In some embodiments, the identifier may be determined based on thelandmark being identified as one of a plurality of landmark types. Inother words, the identifier may be the type of the landmark. Thelandmark types include a traffic sign (e.g., a speed limit sign), a post(e.g., a lamppost), a directional indicator (e.g., a high way exit signwith an arrow indicating a direction), a business sign (e.g., arectangular sign such as sign 2205), a reflector (e.g., a reflectivemirror at a curve for safety purposes), a distance marker, etc. Eachtype may be associated with a unique tag (e.g., a numerical value, atext value, etc.), which requires little data storage (e.g., 4 bytes, 8bytes, etc.). When a landmark is recognized as a specific, stored type,the tag corresponding to the type of the landmark may be stored, alongwith other features of the landmark (e.g., size, shape, location, etc.).

Landmarks may be classified into two categories: landmarks that aredirectly relevant to driving, and landmarks that are not directlyrelevant to driving. Landmarks directly relevant to driving may includetraffic signs, arrows on the road, lane markings, traffic lights, stoplines, etc. These landmarks may include a standard form. Landmarksdirectly relevant to driving may be readily recognizable by theautonomous vehicle as a certain type. Thus, a tag corresponding to thetype of landmarks may be stored with small data storage space (e.g., 1byte, 2 bytes, 4 bytes, 8 bytes, etc.). For example, a tag having anumerical value of “55” may be associated with a stop sign, “100”associated with a speed limit, “108” associated with a traffic light,etc.

Landmarks not directly relevant to driving may include, for example,lampposts, directional signs, businesses signs or billboards (e.g., foradvertisements). These landmarks may not have a standard form. Landmarksthat are not directly relevant to driving, such as billboards foradvertisement and lamppost, may not be readily recognizable by theautonomous vehicle. Signs like billboards may be referred to as generalsigns. General signs may be identified using a condensed signaturerepresentation (or a condensed signature). For example, the identifierassociated with a general sign landmark may include the condensedsignature representation derived from an image of the landmark. Thegeneral sign landmark may be stored using data representing thecondensed signature, rather than an actual image of the landmark. Thecondensed signature may require small data storage space. In someembodiments, the condensed signature may be represented by one or moreinteger numbers, which may require only a few bytes of data storage. Thecondensed signature representation may include unique features,patterns, or characteristics extracted or derived from an image of thelandmarks. The condensed signature representation of landmarks mayindicate an appearance of the landmarks.

The identifier of the landmarks may be stored within the autonomousvehicle road navigation model or sparse map 800, which may be used forproviding navigation guidance to autonomous vehicles. For example, whenanother vehicle later travels along road segment 2200 a previouslydetermined position for the recognized landmark may be used in adetermination of the location of that vehicle relative to a targettrajectory for a road segment.

FIG. 23 illustrates an example environment including a system foridentifying a landmark for use in autonomous vehicle navigation.Vehicles 2301 and 2302 (which may be autonomous vehicles) may travel onroad segment 2200. Vehicles 2301 and 2302 may be similar to othervehicles (e.g., vehicles 200, 2201, and 2202) disclosed in otherembodiments. Vehicle 2301 may include a camera 122, a processor 2310,and a communication unit 2315. Vehicle 2302 may include a camera 122, aprocessor 2311, and a communication unit 2315. In this embodiment, oneof the vehicles 2301 and 2302 may function as a hub vehicle (e.g.,vehicle 2301), which may perform functions performed by server 2230 inthe embodiments shown in FIG. 22. For example, a server similar toserver 2230 may be installed on hub vehicle 2301 to perform functionssimilar to those performed by server 2230. As another example, theprocessor 2310 provided on vehicle 2301 may perform some or all of thefunctions of server 2230.

As shown in FIG. 23, vehicles 2301 and 2302 may communicate with eachother through communication units 2315, 2316, and a communication path2340. Other autonomous vehicles on road segment 2200, although not shownin FIG. 23, may also communicate with hub vehicle 2301. Vehicle 2302(and other vehicles) may transmit landmark data (e.g., images of alandmark 2206) captured or processed by processor 2311 on vehicle 2302to processor 2310 on hub vehicle 2301. Vehicle 2302 may also transmitother navigation information (e.g., road geometry) to hub vehicle 2301.In some embodiments, processor 2310 on hub vehicle 2301 may process thelandmark data received from vehicle 2302 to determine an identifierassociated with a landmark detected by vehicle 2302. In someembodiments, processor 2311 on vehicle 2302 may process images todetermine an identifier associated with a landmark, and transmit theidentifier to vehicle 2301. Processor 2310 on hub vehicle 2301 mayassociate the landmark with road segment 2200, and update an autonomousvehicle road navigation model and/or sparse map 800 to include theidentifier associated with the landmark 2206. Processor 2310 on hubvehicle 2301 may distribute the updated autonomous vehicle roadnavigation model and/or sparse map 800 to a plurality of autonomousvehicles, such as vehicle 2302 and other autonomous vehicles travellingon road segment 2200. It should be understood that any functionsreferenced or described relative to the hub vehicle may be performed byone or more servers located remotely with respect to the vehiclestraveling on a system of roads. For example, such servers may be locatedin one or more central facilities and may be in communication withdeployed vehicles via wireless communication interfaces.

FIG. 24 illustrates a method of determining a condensed signaturerepresentation of a landmark. The condensed signature representation (orcondense signature, or signature) may be determined for a landmark thatis not directly relevant to driving, such as a general sign. Forexample, condensed signature representation may be determined for arectangular business sign (advertisement), such as sign or landmark2205. The condensed signature, rather than an actual image of thegeneral sign may be stored within the model or sparse map, which may beused for later comparison with a condensed signature derived by othervehicles. In the embodiment shown in FIG. 24, an image of the landmark2205 may be mapped to a sequence of numbers of a predetermined datasize, such as 32 bytes (or any other size, such as 16 bytes, 64 bytes,etc.). The mapping may be performed through a mapping function indicatedby arrow 2405. Any suitable mapping function may be used. In someembodiments, a neural network may be used to learn the mapping functionbased on a plurality of training images. FIG. 24 shows an example array2410 including 32 numbers within a range of −128 to 127. The array 2410of numbers may be an example condensed signature representation oridentifier of landmark 2205.

FIG. 25 illustrates another method of determining a condensed signaturerepresentation of a landmark. For example, a color pattern may beextracted or derived from an image of a general sign, such asrectangular business sign 2205. As another example, a brightness patternmay be extracted or derived from the image of the general sign. Thecondensed signature representation may include at least one of the colorpattern or the brightness pattern. In some embodiments, an image of thelandmark 2205 may be divided into a plurality of pixel sections, asshown by the grids in FIG. 25. For each pixel section, a color value ora brightness value may be calculated and associated with the pixelsection, as represented by one of the circle, star, or triangle. Apattern 2500 may represent a color pattern (in which case each of thecircle, start, and triangle represents a color value), or a brightnesspattern (in which case each of the circle, start, and trianglerepresents a brightness value). Pattern 2500 may be used as thecondensed signature representation of landmark 2205.

FIG. 26 illustrates an example block diagram of a memory, which maystore computer code or instructions for performing one or moreoperations for identifying a landmark for use in autonomous vehiclenavigation. As shown in FIG. 26, memory 2600 may store one or moremodules for performing the operations for identifying a landmark for usein autonomous vehicle navigation.

For example, memory 2600 may include a model updating and distributionmodule 2605 and a landmark identifier determination module 2610. In someembodiments, the model updating and distribution module 2605 and thelandmark identifier determination module 2610 may be stored in the samememory 2600, or in different memories. A processor may execute themodules to perform various functions defined by the instructions orcodes included within the modules. For example, when executed by aprocessor, the model updating and distribution module 2605 may cause theprocessor to update an autonomous vehicle road navigation model relativeto a corresponding road segment to include at least one identifierassociated with a landmark. The model updating and distribution module2605 may also cause the processor to distribute the updated autonomousvehicle road navigation model to a plurality of autonomous vehicles forproviding autonomous navigation. When executed by a processor, thelandmark identifier determination module 2610 may cause the processor toanalyze at least one image representative of an environment of a vehicleto identify a landmark in the image. The landmark identifierdetermination module 2610 may also cause the processor to analyze theimage to determine at least one identifier associated with the landmark.The identifier may be used for updating the model in the model updatingand distribution module 2605.

In some embodiments, the landmark identifier determination module 2610may be configured with a certain predefined detection priority. Forexample, the landmark identifier determination module 2610 may cause theprocessor to first search for road signs, and if no road sign is foundwithin a certain distance from a previous landmark, then landmarkidentifier determination module 2610 may use other landmarks.

In addition the landmark identifier determination module 2610 mayinclude a minimum landmark density/frequency and a maximum landmarkdensity/frequency, to limit the landmark frequency (e.g., detected orstored landmarks over a predetermined distance). In some embodiments,these limits may ensure that there are enough landmarks but not too manythat are recognized or detected and stored.

In some embodiments, the landmark density/frequency may be associatedwith a storage size or a bandwidth size. When more road signs areavailable, more storage space or bandwidth may be used. Alternatively oradditionally, different settings may be associated with different typesof landmarks. For example, traffic signs may be associated with a higherlandmark density/frequency, whereas general signs may be associated witha lower landmark density/frequency, such that within a predetermineddistance, more traffic signs may be detected and stored than generalsigns.

In some embodiments, memory 2600 may be included in server 2230, forexample, as part of storage device 2233. Processor 2232 included inserver 2230 may execute the model updating and distribution module 2605to update the autonomous vehicle road navigation model to include atleast one identifier associated with a landmark, and distribute theupdated model to a plurality of autonomous vehicles. In someembodiments, processor 2232 included in server 2230 may receive data(images of landmarks, navigation information, road information, etc.)from vehicles (e.g., 2201, 2202), and may execute the landmarkidentifier determination module 2610 to determine an identifierassociated with the landmark based on the received data.

In some embodiments, memory 2600 may be a memory provided on a hubautonomous vehicle that performs functions of server 2230. For example,when the hub vehicle is vehicle 2201, processor 2210 may execute themodel updating and distribution module 2605 to update the autonomousvehicle road navigation model to include an identifier associated with alandmark. Processor 2210 may also distribute the updated model to aplurality of other autonomous vehicles travelling on road segment 2200.In some embodiments, processor 2210 of hub vehicle 2201 may receive data(e.g., images of landmarks, navigation information, road information,etc.) from other autonomous vehicles (e.g., vehicle 2202). Processor2210 of hub vehicle 2201 may execute the landmark identifierdetermination module 2610 to determine an identifier of a landmark basedon the data received from other autonomous vehicles. For example,processor 2210 of hub vehicle 2201 may analyze an image of anenvironment of another vehicle to identify a landmark, and to determineat least one identifier associated with the landmark. The identifier maybe used by the model updating and distribution module 2605 in updatingthe model by hub vehicle 2201.

In some embodiments, the model updating and distribution module 2605 andthe landmark identifier determination module 2610 may be stored inseparate memories. For example, the model updating and distributionmodule 2605 may be stored in a memory included in server 2230, and thelandmark identifier determination module 2610 may be stored in a memoryprovided on an autonomous vehicle (e.g., a memory of a navigation systemprovided on vehicles 2201, 2202, 2301, and 2302). A processor providedin server 2230 (e.g., processor 2232) may execute the model updating anddistribution module 2605 to update the model and distribute the updatedmodel to autonomous vehicles. A processor provided in the autonomousvehicles (e.g., processor 2210, 2310, or 2311) may execute the landmarkidentifier determination module 2610 to determine an identifierassociated with a landmark.

FIG. 27 is a flowchart showing an exemplary process 2700 for determiningan identifier of a landmark. Process 2700 may be performed when thelandmark identifier determination module 2610 is executed by aprocessor, e.g., processor 2232 included in server 2230, or processor2210, 2310, and 2311 provided on autonomous vehicles. Process 2700 mayinclude acquiring at least one image of an environment of a host vehicle(step 2710). For example, camera 122 provided on host vehicle 2202 (onwhich camera 122 is hosted) may capture at least one image of theenvironment surrounding vehicle 2202. Processor 2210 provided on vehicle2202 may receive the image from camera 122. Process 2700 may alsoinclude analyzing the image to identify a landmark (step 2720). Forexample, processor 2210 provided on vehicle 2202 may analyze the imagereceived from camera 122 to identify a landmark in the environmentsurrounding vehicle 2202. Process 2700 may also include analyzing theimage to determine at least one identifier associated with the landmark(step 2730). For example, processor 2210 provided on vehicle 2202 mayanalyze the image received from camera 122 to determine at least oneidentifier associated with the landmark. The identifier may include anyobservable characteristic associated with the candidate landmark,including any of those discussed above, among others. For example,observation of such landmarks may be made through visual recognitionbased on analysis of captured images and/or may involve sensing by oneof more sensors (e.g., a suspension sensor), or any other means ofobservation.

Process 2700 may include other operations or steps. For example, inidentifying a landmark from the image of the environment, processor 2210may identify the landmark based on a predetermined type. In determiningthe identifier associated with the landmark, processor 2210 maydetermine a position of the landmark based on GPS signals received byvehicle 2202, or other sensor signals that may be used to determine theposition. Processor 2210 may determine at least one of a shape or sizeof the landmark from the image. Processor 2210 may also determine adistance of the landmark to another landmark as appearing in the image,or in the real world. Processor 2210 may extract or derive a condensedsignature representation as part of the identifier of the landmark.Processor 2210 may determine the condensed signature representationbased on mapping the image of the landmark to a sequence of numbers of apredetermined data size (e.g., 32 bytes, 64 byte, etc.). Processor 2210may determine at least one of a color pattern or a brightness pattern asthe condensed signature representation of the landmark.

FIG. 28 is a flowchart showing an exemplary process for updating anddistributing a vehicle road navigation model based on an identifier.Process 2800 may be performed when the model updating and distributionmodule 2602 is executed by a processor, such as processor 2232 includedin server 2230, or processors 2210, 2310, and 2311 included inautonomous vehicles. Process 2800 may include receiving an identifierassociated with a landmark (step 2810). For example, processor 2232 mayreceive at least one identifier associated with a landmark fromautonomous vehicle 2201 or 2202. Process 2800 may include associatingthe landmark with a corresponding road segment (step 2820). For example,processor 2232 may associate landmark 2206 with road segment 2200.Process 2800 may include updating an autonomous vehicle road navigationmodel to include the identifier associated with the landmark (step2830). For example, processor 2232 may update the autonomous vehicleroad navigation model to include an identifier (including, e.g.,position information, size, shape, pattern) associated with landmark2205 in the model. In some embodiments, processor 2232 may also updatesparse map 800 to include the identifier associated with landmark 2205.Process 2800 may include distributing the updated model to a pluralityof autonomous vehicles (step 2840). For example, processor 2232 maydistribute the updated model to autonomous vehicles 2201, 2202, andother vehicles that travel on road segment 2200 at later times. Theupdate model may provide updated navigation guidance to autonomousvehicles.

Process 2800 may include other operations or steps. For example,processor 2232 included in server 2230 may perform some or all of theprocess 2700 for determining an identifier associated with a landmark.Processor 2232 may receive data (including an image of the environment)related to landmarks from vehicles 2201 and 2202. Processor 2232 mayanalyze the data (e.g., the image) to identify a landmark in the imageand to determine an identifier associated with the image.

The disclosed systems and methods may include other features. Forexample, in some embodiments, the vehicle location along a road segmentmay be determined by a processor on the vehicle or a remote server byintegrating the velocity of the vehicle between two landmarks. Thus, thelandmarks may serve as one dimensional (1D) localization anchors. In themodel, the position of a landmark may be computed based on positionsidentified by multiple vehicles in multiple drives by, e.g., averagingthese positions.

For certain landmarks, such as the general signs, the disclosed systemsstore an image signature (e.g., a condensed signature) rather than anactual image of the landmarks. Some types of landmarks may be detectedwith a relatively high precision, and may be readily used forlocalization (e.g., determining the position of the vehicle). Forexample, a sign that is directly relevant to traffic, such as a circularspeed limit sign with the digits “80” may be readily classified as acertain type and easily detected. On the other hand, a beacon sign(e.g., a rectangular advertisement sign) that invites the driver to anearby restaurant may be harder to find without any false detections.The reason is that it is difficult to learn a model for a very diverseclass of objects (e.g., the advertisement sign may not fall into a knownclass or type). When other easy-to-detect signs are not available,general signs may also be used as landmarks, although they pose somerisk of false detections.

For a landmark that is hard to interpret, the disclosed systemsassociate an appearance signature (or condensed signature, or signature)with it. The signature may be stored in the model (e.g., the road modelor the autonomous vehicle road navigation model), together with thepositional information of the landmark. When the vehicle detects such anobject and matches it to the stored mode, the disclosed systems maymatch the signatures of the landmarks. The signature may not encodeclass information (e.g., class information indicating whether theidentified object is a sign), but rather a “same-not-same” information(e.g., information indicating whether the identified object is the sameas one that has been before, or one that has been stored in the model.

The systems (e.g., the remove server or the processor provided on thevehicle) may learn the image signatures from prior examples. A pair ofimages may be tagged the “same” if and only if they belong to the samespecific object (a particular landmark at a particular position). Insome embodiments, the disclosed systems may learn the signatures using aneural network, such as a Siamese neural network. The signature of thelandmark may require small storage space, such as 10 bytes, 20 bytes, 32bytes, 50 bytes, 100 bytes, etc.

The landmarks may be used for longitudinal localization of a vehiclealong a road. Once the relative distances between landmarks (e.g., afirst landmark and a second landmark spaced apart from the firstlandmark by a certain distance along the road) are estimated with asufficient accuracy, then once when the vehicle passes a landmark, thevehicle may “reset” the identifier position estimation and cancel errorsthat emerge from integration of ego speed.

The system may use the ego speed from either the wheels sensor (e.g., aspeed sensor), or from the Global Navigation Satellite System (GNSS)system. In the first option, the system may learn a calibration factorper vehicle, to cancel inter-vehicles variability.

To localize the position of a camera, the disclosed systems may identifyvisual landmarks. Any object with prominent features that may berepeatedly identified may serve as a visual landmark. On road scenarios,road side signs and traffic signs in particular, frequently serve aslandmarks. Traffic signs usually are associated with a type. Trafficsign of “yield” type, for example, may appear exactly or substantiallythe same all over a particular country. When the disclosed systemsidentify a traffic sign with a type, also known as typed-traffic-sign,the systems may look for this type in the map and establish the cameralocalization when a match is found.

Some traffic signs, however, do not look the same. A common example isthe “directional” traffic signs, which tell the driver which lane goeswhere. Other, more generic, signs may also be used, such as signs of aparticular restaurant or advertisements. The standard traffic signs maybe detected using traffic sign recognition algorithms designed torecognize tens or a few hundreds of signs. These standard signs may bestored in the map using one identification byte and a few bytes forlocalization (e.g., position of the signs).

One way to store generic signs is to store the image of the signs in themap database, and look for that image. This solution, however, mayrequire a large memory footprint. Whereas a typed traffic sign mayrequire a single integer (e.g., four bytes), an image patch with anuntyped-traffic sign may require 256 bytes or more to store even a lowresolution of 32×32 pixel image. The solution provided by the disclosedsystems and methods uses a signature function that maps any given imagepatch showing the sign to a unique sequence of 32 bytes (any other bytesmay also be used, e.g., 50 bytes). Conceptually, the output of thefunction is the signature of the image patch showing the sign. Usingthis signature function, the systems may transform any sign to a“signature,” which may be a sequence of 32 bytes (or 8 integers). Usingthe signature, the system may then look in the map for the location of asign with a similar signature or conversely, look in the image for asignature, which according to the map, should be visible in that area ofthe image.

The signature function may be designed to give similar signatures tosimilar image patches, and different signatures to different imagepatches. The systems may use a deep neural network to learn both thesignature function and a distance function between two signatures. Inthe neural network, the actual size of the sign in the image is notknown. Rectangles of various sizes that may be candidates for signs aredetected in the image. Each rectangle may then be scaled to a uniformsize of, for example, 32×32 pixels, although other sizes may also beused. For training the neural network, similar images of the same signare tagged as the “same,” whereas images for different signs captured inthe same geographic location are tagged as “different.” The imagepatches were all scaled to a uniform size. The systems may use a Siamesenetwork that receives two image patches of 32×32 pixels each and outputsa binary bit: 0 means image patches are not the same and 1 means imagepatches are the same. For example, in the example shown in FIG. 24, thesignature 2410 of landmark 2205 may be stored in the map. The signatureincludes a sequence of integer numbers (first sequence), as shown inFIG. 24. When a vehicle passes a sign at the same location as sign 2205,the vehicle may capture an image of the sign, and derive a secondsequence of numbers using the mapping function. For example, the secondsequence of number may include [44, 4, −2, −34, −58, 5, −17, −106, 26,−23, −8, −4, 7, 57, −16, −9, −11, 1, −26, 34, 18, −21, −19, −52, 17, −6,4, 33, −6, 9, −7, −6]. The system may compare the second sequence withthe first second sequence using the neural network, which may output ascore for the comparison. In some embodiments, a negative score mayindicate that the two signatures are not the same, and a positive scoremay indicate that the two signatures are the same. It is noted that thesystem may not require two signatures to be exactly the same in orderfor them to be regarded as the same. The neural network may be capableof processing low resolution input images, which leads to a lowcomputational cost while achieving high performance.

After the training is completed, the Siamese network may be separatedinto two networks: Signature network, which may be the part of thenetwork that receives a single patch, and outputs the “signature” of thelandmark image; and the SNS (same-not-same) network, which may be thepart that receives the two different signatures and outputs a scalar(e.g., the score).

The signature of a landmark may be attached to its location on the map.When a rectangle candidate for landmark is observed, its signature maybe computed using the Signature-network. Then the two signatures, theone from the map and the one from the current landmark are fed into theSNS network. If the output score of the SNS network is negative, it mayindicate that the landmark in the captured image is not the same as theone stored in the map. If the output score of the SNS network ispositive, it may indicate that the landmark in the captured image is thesame as the one stored in the map.

The signatures may require small storage space. For example, thesignatures may use 32 bytes (although other size, such as 10 bytes, 20bytes, 50 bytes, etc., may also be used). Such small-sized signaturesmay also enable transmission on low bandwidth communication channels.

Signatures may be associated with other sizes. There may be a tradeoffbetween the length (hence the size) of the signature and thediscrimination ability of the algorithm. Smaller size may give a highererror rate, whereas larger signature may give less error. Since thedisclosed systems may limit the discrimination requirements to landmarksignatures from the same geographic location, the signature size may bemore compact.

An example use of landmarks in an autonomous navigation system includedin a vehicle is provided below. A camera provided on the vehicle maydetect a landmark candidate, e.g., a rectangular sign. A processor(provided on the vehicle or on a remote server) may scale therectangular sign to a standard size (for example 32×32 pixels). Theprocessor may compute a signature (for example using a system, such as aneural network trained on example data). The processor may compare thecomputed signature to a signature stored in the map. If signatures matchthen the processor may obtain the size of the landmark size from themap. The processor may also estimate a distance from the vehicle to thelandmark based on the landmark size and/or vehicle motion data (e.g.,speed, translation and/or rotation data). The processor may use thedistance from the landmark to localize the position of the vehicle alongthe road or path (e.g., along a target trajectory stored in the map).

The disclosed systems and methods may detect typical street structuressuch as lampposts. The system may take into account both the local shapeof the lamppost and the arrangement of the lamppost in the scene:lampposts are typically at the side of the road (or on the divider),lampposts often appear more than once in a single image and at differentsizes, Lampposts on highways may have fixed spacing based on countrystandards (e.g., around 25 m to 50 m spacing). The disclosed systems mayuse a convolutional neural network algorithm to classify a constantstrip from the image (e.g., 136×72 pixels) that may be sufficient tocatch almost all the street poles. The network may not contain anyaffine layers, and may only be composed of convolution layers, MaxPooling vertical layers and ReLu layers. The network's output dimensionmay be 3 times of the strip width, these three channels may have 3degrees of freedom for each column in the strip. The first degree offreedom may indicate whether there is a street pole in this column, thesecond degree of freedom may indicate this pole's top, and the thirddegree of freedom may indicate its bottom. With the network's outputresults, the system may take all the local maximums that are above athreshold, and built rectangles bounding the poles.

After the system obtains the initial rectangles, the system may use twoalignment neural networks and one filter neural network, and algorithmsto track these poles including optical flow and Kalman Filter. Polesthat were detected multiple times and tracked well were given a higherconfidence.

This disclosure introduces an idea related to landmark definition withinthe context of camera (or vehicle) localization in urban scenarios. Somelandmarks such as traffic signs tend to be quite common and one singlelandmark of this sort may not be uniquely identified unless the GPSlocalization is good. However a sequence of common landmarks may bequite unique and may give localization even when GPS signal is poor asin “urban canyons.” An urban area with high buildings may causesatellites signal to reflect, hence causing poor GPS signals. On theother hand, an urban area is crowded with landmarks of all kinds andsorts. Hence a camera may be used to self-localize, using visuallandmarks. Certain landmarks, however, may be seen repeatedly along thepath or trajectory, making it hard to match the landmark to a concretelocation on the map. A “yield” sign may be common in urban scenarios.When observing just a “yield” traffic sign, the system may not be ableto use it for localization, since there are many “yield” signs in thevicinity of the vehicle, and the system may not be able to know whichone of them is the one captured by the camera.

The disclosed systems and methods may use any of the followingsolutions. In solution one, while virtually all localization algorithmsuse landmarks, the disclosed system may use the positional arrangementof the landmarks to create a positional-landmark. For example, thepositional relation between a plurality of landmarks appearing in thesame image may be measured by a vehicle. The configuration of thelandmarks' positional relation may be taken as a positional-landmark.Instead of just noting the landmarks, the system may compute also thedistances among the different landmarks appearing in the same image.These set of distances may establish a signature of the landmarkpositioning with respect to each other. For example, a sequence oflandmarks detected by the vehicle may be spaced apart by 11 mm betweenthe first and the second landmarks, 16 mm between the second and thirdlandmarks, and 28 mm between the third and fourth landmarks. In someembodiments, the specific positional arrangement of the currentlyvisible landmarks may be unique in the map, and therefore, may be usedas a positional-landmark for localization purposes. Since thepositional-landmarks may be unique, it may be easy to associate themwith a location on the map.

Solution two uses another way to create a unique landmark based onlandmarks is to use the sequence of the landmarks, rather than a singlelandmark. For example, a sequence of landmarks may include a stop sign,a speed limit sign, and a yield sign. While the yield sign landmark maybe abundant, and hence have little localization value, the sequence ofseveral landmarks may be more unique and may lead to unambiguouslocalization.

In some embodiments, the above solutions may be used together. Forexample, the route may be tagged with a sequence of landmarks and thedistance between them. When the GPS signal is weak, the location anddistance between landmarks may be based primarily on odometry (e.g.,based on images and/or inertial sensors and speedometer). Multiplevehicles drive along the route and capture landmarks and their positionsalong the route. The collected information regarding the landmarks maybe sent from the vehicles to the server. The server may collate thelandmark information into landmarks sequences. Each data collectingvehicles may give slightly different distances. The average distance ora robust statistic such as median may be used. The variance among thedistances may also be stored in the map. The sequences of the landmarksmay be aligned taking into account possibly missing landmarks in therecordings from some of the vehicles. The number of times a landmark ismissing gives an indication as to the landmark visibility. Thevisibility of the landmark at that position in the sequence may also bestored in the map.

When a client vehicle drives along the route, it may compare thelandmarks and distances detected by the client vehicle with thesequences stored in the map (or alternatively received from the server).The vehicle may match landmarks types in both sequences, and maypenalize the detected sequence for missing landmarks and for distanceerrors. Landmarks that have low visibility or that have large distancevariances may be penalized less.

The camera for detecting landmarks may be augmented any distancemeasurement apparatus, such as laser or radar.

It is possible that vehicles traveling along a route may record twodifferent sequences. For example, of 50 vehicles traveling along aroute, 20 of them may report a sequence of “star, star, square, circle,star” (where “star,” “square,” “circle” may each represent a certaintype of sign) with consistent distances, whereas the other 30 of themmay report: “star, star, square, square, triangle” with consistentdistances and where the first three “star, star, square” have consistentdistance with the other 20 vehicles. This may indicate that there issome interesting road feature such as an intersection or road split.

Refining Landmark Positions

While the models used for steering in the system need not be globallyaccurate, consistent with disclosed embodiments, global localization maybe useful for navigation systems. For example, global coordinates may beuseful as an index to determine which local map may be relevant fornavigation along a particular road segment or to differentiate onesimilar landmark from another (e.g., a speed limit sign located nearmilepost 45 versus a similar speed limit sign located at milepost 68).Global coordinates may be assigned to landmarks in the model by firstdetermining, based on image analysis, a location of a particularlandmark relative to a host vehicle. Adding these relative coordinatesto the host vehicle's global position may define the global position ofthe landmark. This measurement, however, may be no more accurate thanthe measured position of the host vehicle based on the standardautomotive Global Navigation Satellite System (GNSS) receiver. Thus,while such position determination may be sufficient for indexingpurposes, the disclosed navigational techniques described in detail inlater sections rely upon landmarks to determine a current position of avehicle relative to a target trajectory for a road segment. Usage oflandmarks for this purpose may require more accurate positioninformation for the landmarks than a GPS based measurement can provide.For example, if a GPS measurement is accurate only to +−5 meters, thenthe position determination relative to a target trajectory could beincorrect by 5 or meters, which may be unsuitable for enabling thevehicle to follow a target trajectory.

One solution would be to survey the landmarks associated with a roadsegment and define highly accurate positions for those landmarks inglobal coordinates. Such a method, however, would be prohibitivelycostly in time and money. As another approach to refining the accuracyof a determined landmark position (to a level sufficient to serve as aglobal localization reference for the disclosed methods of autonomousvehicle navigation), multiple measurements of the landmark position maybe made, and the multiple measurements may be used to refine thedetermined position of the landmark. The multiple measurements may beobtained by passing vehicles equipped to determine a position of thelandmark relative to GPS positions for the vehicles obtained as thevehicles pass by the landmark.

FIGS. 22 and 23 each show an example system for identifying a landmarkfor use in autonomous vehicle navigation. The systems may also determinea location or position of the landmark. In the embodiment shown in FIG.22, the system may include server 2230, which may be configured tocommunicate with a plurality of vehicles (e.g., vehicles 2201 and 2202)travelling on road segment 2200. Along the road segment 2200, there maybe one or more landmarks. The landmarks may include at least one of atraffic sign, an arrow, a lane marking, a dashed lane marking, a trafficlight, a stop line, a directional sign, a landmark beacon, or alamppost. For illustration, FIG. 22 shows two landmarks 2205 and 2206.Server 2230 may receive data collected by vehicles 2201 and 2202,including landmarks (e.g., 2205 and 2206) recognized by vehicles 2201and 2202. Data collected by vehicles 2201 and 2202 regarding landmarksmay include position data (e.g., location of the landmarks), physicalsize of the landmarks, distances between two sequentially recognizedlandmarks along road segment 2200, the distance from vehicle 2201 or2202 to a landmark (e.g., 2205 or 2206). Vehicles 2201 and 2202 may bothpass landmark 2206, and may measure positions of landmark 2206 andtransmit the measured positions to server 2230. Server 2230 maydetermine a refined position of a landmark based on the measuredposition data of the landmarks received from vehicles 2201 and 2202. Forexample, the refined position may be an average of the measured positiondata received from vehicles 2201 and 2202, which both pass and recognizelandmark 2206. The refined position of landmark 2206 may be stored in anautonomous vehicle road navigation model or sparse map 800, along withan identifier (e.g., a type, size, condensed signature) of landmark2206. The target position of landmark 2206 may be used by other vehicleslater traveling along road segment 2200 to determine their locationalong a target trajectory associated with road segment 2200, which maybe stored in the model or sparse map. A refined position of a recognizedlandmark (e.g., one that has been included in sparse map 800) may beupdated or further refined when server 2230 receives new measuredposition data from other vehicles relative to the recognized landmark.

In the embodiment shown in FIG. 23, the system may utilize one of theautonomous vehicles as a hub vehicle to perform some or all of thefunctions performed by the remote server 2230 shown in FIG. 22, andtherefore, may not include a server. The hub vehicle may communicatewith other autonomous vehicles and may receive data from other vehicles.The hub vehicle may perform functions related to generating a roadmodel, an update to the model, a sparse map, an update to the sparsemap, a target trajectory, etc. In some embodiments, the hub vehicle mayalso determine a refined position of a landmark stored in a model orsparse map based on multiple positions measured by multiple vehiclestraversing road segment 2200.

For example, in the embodiment shown in FIG. 23, vehicle 2201 may be thehub vehicle, which includes at least one processor (e.g., processor2310) configured to receive various data, including measured position oflandmark 2206, from vehicle 2202. Vehicle 2201 may determine a refinedposition of landmark 2206 based on the measured position data receivedfrom vehicle 2202, and other previously received measured position datafrom vehicles previously passed and recognized landmark 2206. Therefined position of landmark 2206 may be stored within the road model orsparse map 800. The refined position may be updated or refined whenvehicle 2201 receives new measured position data from other vehiclesregarding the same landmark.

FIG. 29 shows an example block diagram of a system 2900 fordetermining/processing/storing a location of a landmark. System 2900 maybe implemented in server 2230 or in a hub vehicle (e.g., 2201). System2900 may include a memory 2910. Memory 2910 may be similar to othermemories disclosed in other embodiments. For example, memory 2910 may bea non-transitory flash memory. Memory 2910 may store data such ascomputer codes or instructions, which may be executed by a processor.System 2900 may include a storage device 2920. Storage device 2920 mayinclude one or more of a hard drive, a compact disc, a magnetic tape,etc. Storage device 2920 may be configured to store data, such as sparsemap 800, an autonomous vehicle road navigation model, road profile data,landmarks information, etc. System 2900 may include at least oneprocessor 2930 configured to execute various codes or instructions toperform one or more disclosed methods or processes. Processor 2930 maybe similar to any other processors disclosed in other embodiments.Processor 2930 may include both hardware components (e.g., computingcircuits) and software components (e.g., software codes). System 2900may also include a communication unit 2940 configured to communicatewith autonomous vehicles via wireless communications, such as wirelessinternet, cellular communications network, etc.

FIG. 30 shows an example block diagram of memory 2910 included in system2900. Memory 2910 may store computer code or instructions for performingone or more operations for determining a location or position of alandmark for use in autonomous vehicle navigation. As shown in FIG. 30,memory 2910 may store one or more modules for performing the operationsfor determining the location of a landmark.

For example, memory 2910 may include a landmark identification module3010 and a landmark position determination module 3020. Memory 2910 mayalso include a landmark position refining module 3030. A processor(e.g., processor 2930) may execute the modules to perform variousfunctions defined by the instructions or codes included within themodules.

For example, when executed by a processor, the landmark identificationmodule 3010 may cause the processor to identify a landmark from an imagecaptured by a camera provided on a vehicle. In some embodiments, theprocessor may acquire at least one environmental image associated with ahost vehicle from a camera installed on the host vehicle. The processormay analyze the at least one environmental image to identify thelandmark in the environment of the host vehicle. The processor mayidentify a type of the landmark, a physical size of the landmark, and/ora condensed signature of the landmark.

When executed by a processor, the landmark position determination module3020 may cause the processor to determine a position of a landmark. Insome embodiments, the processor may receive global positioning system(GPS) data representing a location of the host vehicle, analyze theenvironmental image to determine a relative position of the identifiedlandmark with respect to the host vehicle (e.g., a distance from thevehicle to the landmark). The processor may further determine a globallylocalized position of the landmark based on at least the GPS data andthe determined relative position. This globally localized position maybe used as the location of the landmark and stored in the model or map.

When executed by a processor, the landmark position refining module 3030may refine a position determined by module 3020. In some embodiments,the processor may receive multiple positions relating to the samelandmark from multiple vehicles in multiple drive, or may measure thepositions of the same landmark by the same vehicle in multiple drives.The multiple positions may be used to refine a position of the landmarkalready stored in the map. For example, the processor may calculate anaverage of the multiple positions, or a median value of the multiplepositions and use that (average or median) to update the position of thelandmark stored in the map. As another example, whenever the processorreceives a new position measured by a new vehicle identifying the samelandmark, the new position may be used to update the position of thelandmark already stored in the map.

Various methods may be used to determine the relative position of theidentified landmark with respect to the vehicle based on the analysis ofone or more images captured by a camera provided on the vehicle. Forexample, FIG. 31 shows a method for determining a relative position ofthe landmark to the host vehicle (or a distance from the host vehicle tothe landmark) based on a scale associated with one or more images of thelandmark. In this example, camera 122 provided on vehicle 2201 maycapture an image 3100 of the environment in front of vehicle 2201. Theenvironment may include landmark 3130, which is a speed limit sign, asrepresented by the circle with number “70.” The focus of expansion isindicated by number 3120. Camera 122 may capture a plurality of imagesof the environment, such as a sequence of images. The speed limit sign3130 may appear in a first image at the location indicated by time t1.The speed limit sign 3130 may appear in a second image captured afterthe first image at the location indicated by time t2. The distancebetween the first location (at time t1) to the focus of expansion 3120is indicated by r, and the distance between the first and secondlocations of the speed limit sign 3130 is indicated by d. The distancefrom vehicle 2201 to landmark 3130 may be calculated by Z=V*(t2−t1)*r/d,where V is the speed of vehicle 2201, and Z is the distance from vehicle2201 to landmark 3130 (or the relative position from the landmark 3130to vehicle 2201).

FIG. 32 illustrates a method for determining the relative position ofthe landmark with respect to the host vehicle (or a distance from thevehicle to the landmark) based on an optical flow analysis associatedwith a plurality of images of the environment within a field of view3200. For example, camera 122 may capture a plurality of images of theenvironment in front of vehicle 2201. The environment may include alandmark. A first image of the landmark (represented by the smaller boldrectangle) is referenced as 3210, and a second image of the landmark(represented by the larger bold rectangle) is referenced as 3220. Anoptical flow analysis may analyze two or more images of the same object,and may derive an optical flow field 3230, as indicated by the field ofarrows. The first focus of expansion is referenced by number 3240, andthe second focus of expansion is referenced by number 3250. In someembodiments, the optical flow analysis may determine a time to collision(TTC) based on a rate of expansion derived from the optical flow of theimages. The distance from the vehicle to the landmark may be estimatedbased on the time to collision and the speed of the vehicle.

FIG. 33A is a flowchart showing an example process 3300 for determininga location of a landmark for use in navigation of an autonomous vehicle.Process 3300 may be performed by processor 2930, which may be includedin a remote server (e.g., server 2230), or on an autonomous vehicle(e.g., vehicle 2301). Process 3300 may include receiving a measuredposition of a landmark (step 3310). For example, processor 2930 mayreceive a measured position of landmark 2206 from vehicle 2202. Vehicle2202 may measure the position of the landmark 2206 based on the GPS dataindicating the location of the vehicle, a relative position of thelandmark with respect to vehicle 2202 determined from analysis of one ormore images of an environment of vehicle 2202 including the landmark.Process 3000 may include determining a refined position of the landmarkbased on the measured position and at least one previously acquiredposition of the landmark (step 3320). For example, processor 2930 mayaverage the measured position with the at least one previously acquiredposition of the landmark, such as one or more previously acquiredpositions received from other vehicles that identified the landmark. Insome embodiments, processor 2930 may average the measured position witha position stored in a map (e.g., sparse map 800) that is determinedbased on at least one previously acquired position of the landmark.Processor 2930 may use the averaged position as the refined position. Insome embodiments, processor 2930 may calculate a median value of themeasured position and the at least one previously acquired position(e.g., a plurality of previously acquired positions), and use the medianvalue as the refined position. Other statistical parameters that may beobtained from the measured position and the plurality of previouslyacquired positions may be used as the target position. Process 3000 mayupdate the location of the landmark stored in a map with the refinedposition (step 3330). For example, processor 2930 may replace theposition stored in the map with the refined position. When new positiondata is received, processor 2930 may repeat steps 3320 and 3330 torefine the position of the landmark stored in the map, therebyincreasing the accuracy of the position of the landmark.

FIG. 33B is a flowchart showing an example process 3350 for measuringthe position of a landmark. Process 3350 may be performed by processor2930, which may be provided in server 2230 or the autonomous vehicles(e.g., vehicles 2201, 2202, 2301, and 2302). A previously acquiredposition stored in a map may also be obtained using process 3350.Process 3350 may include acquiring an environmental image from a camera(step 3351). For example, camera 122 provided on vehicle 2202 maycapture one or more images of the environment of vehicle 2202, which mayinclude landmark 2206. Processor 2930 may acquire images from camera122. Process 3350 may include analyzing the environmental image toidentify a landmark (step 3351). For example, processor 2930 (orprocessor 2210) provided on vehicle 2202 may analyze the images toidentify landmark 2206. Process 3350 may also include receiving GPS datafrom a GPS unit provided on the vehicle (step 3353). For example,processor 2930 may receive GPS data from the GPS unit provided onvehicle 2202. The GPS data may represent the location of vehicle 2202(host vehicle). Process 3350 may include analyzing the environmentalimage to determine a relative position of the identified landmark withrespect to the vehicle (step 3354). For example, processor 2930 mayanalyze the images of the environment to determine a relative positionof the identified landmark 2206 with respect to vehicle 2202 using asuitable method. In some embodiments, processor 2930 may analyze theimages to determine the relative position based on a scale discussedabove in connection with FIG. 31. In some embodiments, processor 2930may analyze the images to determine the relative position based on anoptical flow analysis of the images, as discussed above in connectionwith FIG. 32. Process 3350 may further include determining a globallylocalized position of the landmark based on the GPS data and thedetermined relative position of the landmark with respect to the vehicle(step 3355). For example, processor 2930 may calculate the globallylocalized position of the landmark by combining the position of vehicle2202 as indicated by the GPS data and the relative position (or distancefrom vehicle 2202 to landmark 2206). The globally localized position ofthe landmark may be used as the measured position of the landmark.

The disclosed systems and methods may include other features discussedbelow. The disclosed system may be capable of steering an autonomousvehicle along a target trajectory without knowing the precise locationof the vehicle relative to a global coordinate frame. The GPSinformation may have an error of greater than 10 m, so the GPSinformation is primarily used to index the memory in order to retrieve alandmark candidate or a relevant road tile. The global localization maybe determined using the visual ego motion. In order to avoid drifts, thesystem may estimate the GPS location of the landmarks by combining theGPS position of the host vehicle and the relative position of thelandmark to the host vehicle. The global landmark location may berefined (e.g., averaged) with location data obtained from multiplevehicles and multiple drives. The measured position or location of thelandmark may behave like a random variable, and hence may be averaged toimprove accuracy. The GPS signals are used primarily as a key or indexto a database storing the landmarks, and do not have to have highprecision for determining the position of the vehicle. Low precision GPSdata may be used to determine the location of the vehicle, which is usedto determine the position of the landmark. Errors introduced by the lowprecision GPS data may accumulate. Such errors may be fixed by averagingthe position data of the landmark from multiple drives.

In some embodiments, for steering purpose, the GPS coordinates may onlybe used to index the database. The GPS data may not be taken intoaccount in the computation of the steering angle. The model includingthe location of the landmarks may be transitioned to a global coordinatesystem. The transition may include determining the UPS coordinates oflandmarks by averaging, concluding the GPS position of the vehicle near(globally localized) landmarks, and extending the global localizationaway from landmarks, by using the curve geometry, the location alongpath, the lane assignment and the in-lane position.

Autonomous Navigation Using a Sparse Road Model

In some embodiments, the disclosed systems and methods may use a sparseroad model for autonomous vehicle navigation. For example, the disclosedsystems and methods may provide navigation based on recognizedlandmarks, align a vehicle's tail for navigation, allow a vehicle tonavigate road junctions, allow a vehicle to navigate using localoverlapping maps, allow a vehicle to navigate using a sparse map,navigate a vehicle based on an expected landmark location, autonomouslynavigate a vehicle based on road signatures, navigate a vehicle forwardbased on a rearward facing camera, navigate a vehicle based on a freespace determination, and navigate a vehicle in snow. Additionally, thedisclosed embodiments provide systems and methods for autonomous vehiclespeed calibration, determining a lane assignment o of a vehicle based ona recognized landmark location, and using super landmarks as navigationaids when navigating a vehicle. These systems and methods are detailedbelow.

Navigation Based on Recognized Landmarks

Consistent with disclosed embodiments, the system may use landmarks, forexample, to determine the position of a host vehicle along a pathrepresentative of a target road model trajectory (e.g., by identifyingan intersection point of a relative direction vector to the landmarkwith the target road model trajectory). Once this position isdetermined, a steering direction can be determined by comparing aheading direction to the target road model trajectory at the determinedposition. Landmarks may include, for example, any identifiable, fixedobject in an environment of at least one road segment or any observablecharacteristic associated with a particular section of the road segment.In some cases, landmarks may include traffic signs (e.g., speed limitsigns, hazard signs, etc.). In other cases, landmarks may include roadcharacteristic profiles associated with a particular section of a roadsegment. In yet other cases, landmarks may include road profiles assensed, for example, by a suspension sensor of the vehicle. Furtherexamples of various types of landmarks are discussed in previoussections, and some landmark examples are shown in FIG. 10.

FIG. 34 illustrates vehicle 200 (which may be an autonomous vehicle)travelling on road segment 3400 in which the disclosed systems andmethods for navigating vehicle 200 using one or more recognizedlandmarks 3402, 3404 may be used. Although, FIG. 34 depicts vehicle 200as equipped with image capture devices 122, 124, 126, more or fewerimage capture devices may be employed on any particular vehicle 200. Asillustrated in FIG. 34, road segment 3400 may be delimited by left side3406 and right side 3408. A predetermined road model trajectory 3410 maydefine a preferred path (e.g., a target road model trajectory) withinroad segment 3400 that vehicle 200 may follow as vehicle 200 travelsalong road segment 3400. In some exemplary embodiments, predeterminedroad model trajectory 3410 may be located equidistant from left side3406 and right side 3408. It is contemplated however that predeterminedroad model trajectory 3410 may be located nearer to one or the other ofleft side 3406 and right side 3408 of road segment 3400. Further,although FIG. 34 illustrates one lane in road segment 3400, it iscontemplated that road segment 3400 may have any number of lanes. It isalso contemplated that vehicle 200 travelling along any lane of roadsegment 3400 may be navigated using one or more landmarks 3402, 3404according to the disclosed methods and systems.

Image acquisition unit 120 may be configured to acquire an imagerepresentative of an environment of vehicle 200. For example, imageacquisition unit 120 may obtain an image showing a view in front ofvehicle 200 using one or more of image capture devices 122, 124, 126.Processing unit 110 of vehicle 200 may be configured to detect one ormore landmarks 3402, 3404 in the one or more images acquired by imageacquisition unit 120. Processing unit 110 may detect the one or morelandmarks 3402, 3404 using one or more processes of landmarkidentification discussed above with reference to FIGS. 22-28. AlthoughFIG. 34 illustrates only two landmarks 3402, 3404, it is contemplatedthat vehicle 200 may detect fewer than or more than landmarks 3402, 3404based on the images acquired by image acquisition unit 120.

Processing unit 110 may be configured to determine positions 3432, 3434of the one or more landmarks 3402, 3404, respectively, relative to acurrent position 3412 of vehicle 200. Processing unit 110 may also beconfigured to determine a distance between current position 3412 ofvehicle 200 and the one or more landmarks 3402, 3404. Further,processing unit 110 may be configured to determine one or moredirectional indicators 3414, 3416 of the one or more landmarks 3402,3404 relative to current position 3412 of vehicle 200. Processing unit110 may be configured to determine directional indicators 3414, 3416 asvectors originating from current position 3412 of vehicle 200 andextending towards, for example, positions 3432, 3434 of landmarks 3402,3404, respectively.

Processing unit 110 may also be configured to determine an intersectionpoint 3418 of the one or more directional indicators 3414, 3416 withpredetermined road model trajectory 3410. In one exemplary embodiment asillustrated in FIG. 34, intersection point 3418 may coincide withcurrent position 3412 of vehicle 200. This may occur, for example, whenvehicle 200 is located on predetermined road model trajectory 3410.Although generally vehicle 200 may be expected to be located on or verynear predetermined road model trajectory 3410, it is contemplated thatvehicle 200 may not be located on predetermined road model trajectory3410 as will be discussed below with respect to FIG. 35.

Processing unit 110 may be configured to determine a direction 3420 ofpredetermined road model trajectory 3410 at intersection point 3418.Processing unit 110 may determine direction 3420 as a directiontangential to predetermined road model trajectory 3410. In one exemplaryembodiment, processing unit 110 may be configured to determine direction3420 based on a gradient or slope of a three-dimensional polynomialrepresenting predetermined road model trajectory 3410.

Processing unit 110 may also be configured to determine headingdirection 3430 of vehicle 200. As illustrated in FIG. 34, headingdirection 3430 of vehicle 200 may be a direction along which imagecapture device 122 may be oriented relative to a local coordinate systemassociated with vehicle 200. Processing unit 110 may be configured todetermine whether heading direction 3430 of vehicle 200 is aligned with(i.e., generally parallel to) direction 3420 of predetermined road modeltrajectory 3410. When heading direction 3430 is not aligned withdirection 3420 of predetermined road model trajectory 3410 atintersection point 3418, processing unit 110 may determine an autonomoussteering action such that heading direction 3430 of vehicle 200 may bealigned with direction 3420 of predetermined road model trajectory 3410.In one exemplary embodiment, an autonomous steering action may include,for example, a determination of an angle by which the steering wheel orfront wheels of vehicle 200 may be turned to help ensure that headingdirection 3430 of vehicle 200 may be aligned with direction 3420 ofpredetermined road model trajectory 3410. In another exemplaryembodiment, an autonomous steering action may also include a reductionor acceleration in a current velocity of vehicle 200 to help ensure thatheading direction 3430 of vehicle 200 may be aligned with direction 3420of predetermined road model trajectory 3410 in a predetermined amount oftime. Processing unit 110 may be configured to execute instructionsstored in navigational response module 408 to trigger a desirednavigational response by, for example, turning the steering wheel ofvehicle 200 to achieve a rotation of a predetermined angle. Rotation bythe predetermined angle may help align heading direction 3430 of vehicle200 with direction 3420.

Processing unit 110 may include additional considerations whendetermining the autonomous steering action. For example, in someexemplary embodiments, processing unit 110 may determine the autonomoussteering action based on a kinematic and physical model of the vehicle,which may include the effects of a variety of possible autonomoussteering actions on the vehicle or on a user of vehicle 200. Processingunit 110 may implement a selection criteria for selecting at least oneautonomous steering action from the plurality of autonomous steeringactions. In other exemplary embodiments, processing unit 110 maydetermine an autonomous steering action based on a “look ahead”operation, which may evaluate portions of road segment 3400 located infront of current location 3418 of vehicle 200. Processing unit 110 maydetermine an effect of one or more autonomous steering actions on thebehavior of vehicle 200 or on a user of vehicle 200 at a location infront of current location 3418, which may be caused by the one or moreautonomous steering actions. In yet other exemplary embodiments,processing unit 110 may further account for the presence and behavior ofone or more other vehicles in the vicinity of vehicle 200 and a possible(estimated) effect of one or more autonomous steering actions on suchone or more other vehicles. Processing unit 110 may implement theadditional considerations as overrides. Thus, for example, processingunit 110 may initially determine an autonomous steering action that mayhelp ensure that heading direction 3430 of vehicle 200 may be alignedwith direction 3420 of predetermined road model trajectory 3410 atcurrent location 3418. When processing unit 110 determines that thedetermined autonomous steering does not comply with one or moreconstraints imposed by the additional considerations, processing unit110 may modify the autonomous steering action to help ensure that allthe constraints may be satisfied.

Image acquisition unit 120 may repeatedly acquire an image of theenvironment in front of vehicle 200, for example, after a predeterminedamount of time. Processing unit 110 may also be configured to repeatedlydetect the one or more landmarks 3402, 3404 in the image acquired byimage acquisition unit 120 and determine the autonomous steering actionas discussed above. Thus, image acquisition unit 120 and processing unit110 may cooperate to navigate vehicle 200 along road segment 3400 usingone or more landmarks 3402, 3404.

FIG. 35 illustrates another vehicle 200 travelling on road segment 3400in which the disclosed systems and methods for navigating vehicle 200using one or more recognized landmarks 3402, 3404 may be used. UnlikeFIG. 34, vehicle 200 of FIG. 35 is not located on predetermined roadmodel trajectory 3410. As a result, as illustrated in FIG. 35,intersection point 3418 of directional indicator 3416 may not coincidewith current position 3412 of vehicle 200.

As discussed above with respect to FIG. 34, processing unit 110 may beconfigured to determine a direction 3420 of predetermined road modeltrajectory 3410 at intersection point 3418. Processing unit 110 may alsobe configured to determine whether heading direction 3430 of vehicle 200is aligned with (i.e. generally parallel to) direction 3420. Whenheading direction 3430 is not aligned with direction 3420 ofpredetermined road model trajectory 3410 at intersection point 3418,processing unit 110 may determine a first autonomous steering actionsuch that heading direction 3430 of vehicle 200 may be aligned withdirection 3420 of predetermined road model trajectory 3410. For example,as illustrated in FIG. 35, processing unit 110 may determine the firstautonomous steering action to require a rotation by an angle to helpensure that heading direction 3430 of vehicle 200 may be aligned withdirection 3420.

In addition, when current position 3412 of vehicle 200 is not located onpredetermined road model trajectory 3410, processing unit 120 maydetermine a second autonomous steering action to help ensure thatvehicle 200 may move from current position 3412 to intersection point3418 on predetermined road model trajectory 3410. For example, asillustrated in FIG. 35, processing unit 110 may determine a distance “d”by which vehicle 200 must be translated to move current position 3412 tocoincide with intersection point 3418 on predetermined road modeltrajectory 3410. Although not illustrated in FIG. 35, processing unit110 may also be configured to determine a rotation that may be requiredto help ensure that vehicle 200 may move from current position 3412 tointersection point 3418 on predetermined road model trajectory 3410.Processing unit 110 may be configured to execute instructions stored innavigational response module 408 to trigger a desired navigationalresponse corresponding to the first autonomous steering action, thesecond autonomous steering action, or some combination of the first andthe second autonomous steering actions. In some embodiment, processingunit 110 may execute instructions to trigger a desired navigationalresponse corresponding to the first autonomous steering action and thesecond autonomous steering action sequentially in any order.

FIG. 36 is a flowchart showing an exemplary process 3600, for navigatingvehicle 200 along road segment 3400, using one or more landmarks 3402,3404, consistent with disclosed embodiments. Steps of process 3600 maybe performed by one or more of processing unit 110 and image acquisitionunit 120, with or without the need to access memory 140 or 150. Theorder and arrangement of steps in process 3600 is provided for purposesof illustration. As will be appreciated from this disclosure,modifications may be made to process 3600 by, for example, adding,combining, removing, and/or rearranging the steps for the process.

As illustrated in FIG. 36, process 3600 may include a step 3602 ofacquiring an image representative of an environment of the vehicle. Inone exemplary embodiment, image acquisition unit 120 may acquire one ormore images of an area forward of vehicle 200 (or to the sides or rearof a vehicle, for example). For example, image acquisition unit 120 mayobtain an image using image capture device 122 having a field of view202. In other exemplary embodiments, image acquisition unit 120 mayacquire images from one or more of image capture devices 122, 124, 126,having fields of view 202, 204, 206. Image acquisition unit 120 maytransmit the one or more images to processing unit 110 over a dataconnection (e.g., digital, wired, USB, wireless, Bluetooth, etc.).

Process 3600 may also include a step 3604 of identifying one or morelandmarks 3402, 3404 in the one or more images. Processing unit 110 mayreceive the one or more images from image acquisition unit 120.Processing unit 110 may execute monocular image analysis module 402 toanalyze the plurality of images at step 3604, as described in furtherdetail in connection with FIGS. 5B-5D. By performing the analysis,processing unit 110 may detect a set of features within the set ofimages, for example, one or more landmarks 3402, 3404. Landmarks 3402,3404 may include one or more traffic signs, arrow markings, lanemarkings, dashed lane markings, traffic lights, stop lines, directionalsigns, reflectors, landmark beacons, lampposts, a change is spacing oflines on the road, signs for businesses, and the like.

In some embodiments, processing unit 110 may execute monocular imageanalysis module 402 to perform multi-frame analysis on the plurality ofimages to detect landmarks 3402, 3404. For example, processing unit 110may estimate camera motion between consecutive image frames andcalculate the disparities in pixels between the frames to construct a3D-map of the road. Processing unit 110 may then use the 3D-map todetect the road surface, as well as landmarks 3402, 3404. In anotherexemplary embodiment, image processor 190 of processing unit 110 maycombine a plurality of images received from image acquisition unit 120into one or more composite images. Processing unit 110 may use thecomposite images to detect the one or more landmarks 3402, 3404.

In some embodiments, processing unit 110 may be able to recognizevarious attributes of objects that may qualify as potential landmarks.This information may be uploaded to a server, for example, remote fromthe vehicle. The server may process the received information and mayestablish a new, recognized landmark within sparse data map 800, forexample. It may also be possible for the server to update one or morecharacteristics (e.g., size, position, etc.) of a recognized landmarkalready included in sparse data map 800.

In some cases, processing unit 110 may receive information from a remoteserver that may aid in locating recognized landmarks (e.g., thoselandmarks that have already been identified and represented in sparsedata map 800). For example, as a vehicle travels along a particular roadsegment, processor 110 may access one or more local maps correspondingto the road segment being traversed. The local maps may be part ofsparse data map 800 stored on a server located remotely with respect tothe vehicle, and the one or more local maps may be wirelessly downloadedas needed. In some cases, the sparse map 800 may be stored locally withrespect to the navigating vehicle. The local maps may include variousfeatures associated with a road segment. For example, the local maps mayinclude a polynomial spline representative of a target trajectory thatthe vehicle should follow along the road segment. The local maps mayalso include representations of recognized landmarks. In some cases, aspreviously described, the recognized landmarks may include informationsuch as a landmark type, position, size, distance to another landmark,or other characteristics. In the case of non-semantic signs (e.g.,general signs not necessarily associated with road navigation), forexample, the information stored in sparse data map 800 may include acondensed image signature associated with the non-semantic road sign.

Such information received from sparse data map 800 may aid processorunit 110 in identifying recognized landmarks along a road segment. Forexample, processor unit 110 may determine based on its current position(determined, for example, based on GPS data, dead reckoning relative toa last determined position, or any other suitable method) andinformation included in a local map (e.g., a localized position of thenext landmark to be encountered and/or information indicating a distancefrom the last encountered landmark to the next landmark) that arecognized landmark should be located at a position approximately 95meters ahead of the vehicle and 10 degrees to the right of a currentheading direction. Processor unit 110 may also determine from theinformation in the local map that the recognized landmark is of a typecorresponding to a speed limit sign and that the sign has a rectangularshape of about 2 feet wide by 3 feet tall.

Thus, when processor unit 110 receives images captured by the onboardcamera, those images may be analyzed by searching for an object at theexpected location of a recognized landmark from sparse map 800. In thespeed limit sign example, processor unit 110 may review captured imagesand look for a rectangular shape at a position in the image 10 degreesto the right of a heading direction of the vehicle. Further, theprocessor may look for a rectangular shape occupying a number of pixelsof the image that a 2 foot by 3 foot rectangular sign would be expectedto occupy at a relative distance of 95 meters. Upon identifying such anobject in the image, where expected, the processor may develop a certainconfidence level that the expected recognized landmark has beenidentified. Further confirmation may be obtained, for example, byanalyzing the image to determine what text or graphics appear on thesign in the captured images. Through textual or graphics recognitionprocesses, the processor unit may determine that the rectangular shapein the captured image includes the text “Speed Limit 55.” By comparingthe captured text to a type code associated with the recognized landmarkstored in sparse data map 800 (e.g., a type indicating that the nextlandmark to be encountered is a speed limit sign), this information canfurther verify that the observed object in the captured images is, infact, the expected recognized landmark.

Process 3600 may include a step 3606 of determining a current position3412 of vehicle 200 relative to a target trajectory. Processing unit 110may determine current position 3412 of vehicle 200 in many differentways. For example, processing unit 110 may determine current position3412 based on signals from position sensor 130, for example, a GPSsensor. In another exemplary embodiment, processing unit 110 maydetermine current position 3412 of vehicle 200 by integrating a velocityof vehicle 200 as vehicle 200 travels along predetermined road modeltrajectory 3410. For example, processing unit 110 may determine a time“t” required for vehicle 200 to travel between two locations onpredetermined road model trajectory 3410. Processing unit 110 mayintegrate the velocity of vehicle 200 over time t to determine currentposition 3412 of vehicle 200 relative to the two locations onpredetermined road model trajectory 3410.

Once a recognized landmark is identified in a captured image,predetermined characteristics of the recognized landmark may be used toassist a host vehicle in navigation. For example, in some embodiments,the recognized landmark may be used to determine a current position ofthe host vehicle. In some cases, the current position of the hostvehicle may be determined relative to a target trajectory from sparsedata model 800. Knowing the current position of the vehicle relative toa target trajectory may aid in determining a steering angle needed tocause the vehicle to follow the target trajectory (for example, bycomparing a heading direction to a direction of the target trajectory atthe determined current position of the vehicle relative to the targettrajectory).

A position of the vehicle relative to a target trajectory from sparsedata map 800 may be determined in a variety of ways. For example, insome embodiments, a 6D Kalman filtering technique may be employed. Inother embodiments, a directional indicator may be used relative to thevehicle and the recognized landmark. For example, process 3600 may alsoinclude a step 3608 of determining one or more directional indicators3414, 3416 associated with the one or more landmarks 3402, 3404,respectively. Processing unit 110 may determine directional indicators3414, 3416 based on the relative positions 3432, 3434 of the one or morelandmarks 3402, 3404, respectively, relative to current position 3412 ofvehicle 200. For example, processing unit 110 may receive landmarkpositions 3432, 3434 for landmarks 3402, 3404, respectively, frominformation, which may be stored in one or more databases in memory 140or 150. Processing unit 110 may also determine distances between currentposition 3412 of vehicle 200 and landmark positions 3432, 3434 forlandmarks 3402, 3404, respectively. In addition, processing unit 110 maydetermine directional indicator 3414 as a vector extending from currentposition 3412 of vehicle 200 and extending along a straight line passingthrough current position 3412 and landmark position 3432. Likewise,processing unit 110 may determine directional indicator 3416 as a vectorextending from current position 3412 of vehicle 200 and extending alonga straight line passing through current position 3412 and landmarkposition 3434. Although two landmarks 3402, 3404 are referenced in theabove discussion, it is contemplated that processing unit 110 maydetermine landmark positions 3432, 3434, distances between currentposition 3412 and landmark positions 3402, 34014, and directionalindicators 3414, 3416 for fewer than or more than landmarks 3402, 3404.

Process 3600 may include a step 3610 of determining an intersectionpoint 3418 of directional indicator 3416 with predetermined road modeltrajectory 3410. Processing unit 110 may determine a location ofintersection point 3418 at which predetermined road model trajectory3410 intersects with a straight line extending between current position3412 of vehicle 200 and landmark position 3434. Processing unit 110 mayobtain a mathematical representation of predetermined road modeltrajectory 3410 from information stored in memories 140, 150. Processingunit 110 may also generate a mathematical representation of a straightline passing through both current position 3412 of vehicle 200 andlandmark position 3434 of landmark 3404. Processing unit 110 may use themathematical representation of predetermined road model trajectory 3410and the mathematical representation of a straight line extending betweencurrent position 3412 and landmark position 3434 to determine a locationof intersection point 3418.

In one exemplary embodiment as illustrated in FIG. 34, intersectionpoint 3418 may coincide with current position 3412 of vehicle 200 (e.g.,a position of a point of reference, which may be arbitrarily assigned,associated with the vehicle). This may happen, for example, when vehicle200 is located on predetermined road model trajectory 3410. In anotherexemplary embodiment as illustrated in FIG. 35, intersection point 3418may be separated from current position 3412. Processing unit 110 maydetect that vehicle 200 is not located on predetermined road modeltrajectory 3410 by comparing a first distance “D₁” (see, e.g., FIG. 35)between current position 3412 and landmark position 3434 with a seconddistance “D₂” between intersection point 3418 and landmark position3434.

When intersection point 3418 is separated from current position 3412 ofvehicle 200, processing unit 110 may determine an amount of translationand/or rotation that may be required to help move vehicle 200 fromcurrent position 3412 to intersection point 3418 on predetermined roadmodel trajectory 3410. In some exemplary embodiments, processing unit110 may execute navigation module 408 to cause one or more navigationalresponses in vehicle 200 based on the analysis performed at step 520 andthe techniques as described above in connection with FIG. 4. Forexample, processing unit 110 may issue commands to steering system 240to move vehicle 200 so that a current position 3412 of vehicle 200 maycoincide with intersection point 3418.

Process 3600 may include a step 3612 of determining direction 3420 ofpredetermined road model trajectory 3410 at intersection point 3418. Inone exemplary embodiment, processing unit 110 may obtain a mathematicalrepresentation (e.g. three-dimensional polynomial) of predetermined roadmodel trajectory 3410. Processing unit 110 may determine direction 3420as a vector oriented tangentially to predetermined road model trajectory3410 at intersection point 3418. For example, processing unit 110 maydetermine direction 3420 as a vector pointing along a gradient of themathematical representation of predetermined road model trajectory 3410at intersection point 3418.

Process 3600 may also include a step 3614 of determining an autonomoussteering action for vehicle 200. In one exemplary embodiment, processingunit 110 may determine a heading direction 3430 of vehicle 200. Forexample, as illustrated in FIGS. 34 and 35, processing unit 110 maydetermine heading direction 3430 of vehicle 200 as the direction inwhich image capture device 122 may be oriented relative to a localcoordinate system associated with vehicle 200. In another exemplaryembodiment, processing unit 200 may determine heading direction 3430 asthe direction of motion of vehicle 200 at current position 3412.Processing unit 110 may also determine a rotational angle betweenheading direction 3430 and direction 3420 of predetermined road modeltrajectory 3410. Processing unit 110 may execute the instructions innavigational module 408 to determine an autonomous steering action forvehicle 200 that may help ensure that heading direction 3430 of vehicle200 is aligned (i.e., parallel) with direction 3420 of predeterminedroad model trajectory 3410 at intersection point 3418. Processing unit110 may also send control signals to steering system 240 to adjustrotation of the wheels of vehicle 200 to turn vehicle 200 so thatheading direction 3430 may be aligned with direction 3420 ofpredetermined road model trajectory 3410 at intersection point 3418. Inone exemplary embodiment, processing unit 110 may send signals tosteering system 240 to adjust rotation of the wheels of vehicle 200 toturn vehicle 200 until a difference between heading direction 3430 anddirection 3420 of predetermined road model trajectory 3410 atintersection point 3418 may be less than a predetermined thresholdvalue.

Processing unit 110 and/or image acquisition unit 120 may repeat steps3602 through 3614 after a predetermined amount of time. In one exemplaryembodiment, the predetermined amount of time may range between about 0.5seconds to 1.5 seconds. By repeatedly determining intersection point3418, heading direction 3430, direction 3420 of predetermined road modeltrajectory 3410 at intersection point 3418, and the autonomous steeringaction required to align heading direction 3430 with direction 3420,processing unit 110 and/or image acquisition unit 120 may help tonavigate vehicle 200, using the one or more landmarks 3402, 3404, sothat vehicle 200 may travel along road segment 3400.

Tail Alignment Navigation

Consistent with disclosed embodiments, the system can determine asteering direction for a host vehicle by comparing and aligning atraveled trajectory of the host vehicle (the tail) with a predeterminedroad model trajectory at a known location along the road modeltrajectory. The traveled trajectory provides a vehicle heading directionat the host vehicle location, and the steering direction can beobtained, relative to the heading direction, by determining atransformation (e.g., rotation and potentially translation) thatminimizes or reduces an error between the traveled trajectory and theroad model trajectory at the known location of the vehicle along theroad model trajectory.

Tail alignment is a method of aligning an autonomous vehicle's headingwith a pre-existing model of the path based on information regarding thepath over which the vehicle has already travelled. Tail alignment uses atracked path of the autonomous vehicle over a certain distance (hencethe “tail”). The tracked path is a representation of the path over whichthe autonomous vehicle has already travelled in order to reach a currentlocation of the autonomous vehicle. For example, the tracked path mayinclude a predetermined distance (e.g. 60 m or other desired length) ofthe path behind the autonomous vehicle over which the autonomous vehicletravelled to reach its current location. The tracked path may becompared with the model to determine, for example, a heading angle ofthe autonomous vehicle.

In some embodiments, a rear looking camera may be used to determine oraid in determination of the travelled path. A rear looking camera may beuseful both for modeling, heading estimation, and lateral offsetestimation. By adding a rear looking camera it may be possible to boostthe reliability of the system, since a bad illumination situation (e.g.,low sun on the horizon) rarely would affect both front looking and rearlooking cameras.

The tracked path can also optionally be combined with a predicted pathof the autonomous vehicle. The predicted path may be generated byprocessing images of the environment ahead of the autonomous vehicle anddetecting lane, or other road layout, markings. In this regard it isworth noting, that in a potential implementation of the presentdisclosure, a road model may diverge due to accumulated errors(integration of ego motion). Thus, for example, a predicted path over apredetermined distance (e.g. 40 m) ahead of the current location of theautonomous vehicle may be compared with the tracked path to determinethe heading angle for the autonomous vehicle.

FIG. 37 illustrates vehicle 200 (which may be an autonomous vehicle)travelling on road segment 3700 in which the disclosed systems andmethods for navigating vehicle 200 using tail alignment may be used. Asused here and throughout this disclosure, the term “autonomous vehicle”refers to vehicles capable of implementing at least one navigationalchange in course without driver input. To be autonomous, a vehicle neednot be fully automatic (e.g., fully operational without a driver orwithout driver input). Rather, an autonomous vehicle includes those thatcan operate under driver control during certain time periods and withoutdriver control during other time periods. Autonomous vehicles may alsoinclude vehicles that control only some aspects of vehicle navigation,such as steering (e.g., to maintain a vehicle course between vehiclelane constraints), but may leave other aspects to the driver (e.g.,braking). In some cases, autonomous vehicles may handle some or allaspects of braking, speed control, and/or steering of the vehicle.

Although, FIG. 37 depicts vehicle 200 as equipped with image capturedevices 122, 124, 126, more or fewer image capture devices may beemployed on any particular vehicle 200. As illustrated in FIG. 37, roadsegment 3700 may be delimited by left side 3706 and right side 3708. Apredetermined road model trajectory 3710 may define a preferred path(i.e. a target road model trajectory) within road segment 3700 thatvehicle 200 may follow as vehicle 200 travels along road segment 3700.In some exemplary embodiments, predetermined road model trajectory 3710may be located equidistant from left side 3706 and right side 3708. Itis contemplated however that predetermined road model trajectory 3710may be located nearer to one or the other of left side 3706 and rightside 3708 of road segment 3700. Further, although FIG. 37 illustratesone lane in road segment 3700, it is contemplated that road segment 3700may have any number of lanes. It is also contemplated that vehicle 200travelling along any lane of road segment 3400 may be navigated usingtail alignment according to the disclosed methods and systems.

Image acquisition unit 120 may be configured to acquire a plurality ofimages representative of an environment of vehicle 200, as vehicle 200travels along road segment 3700. For example, image acquisition unit 120may obtain the plurality of images showing views in front of vehicle 200using one or more of image capture devices 122, 124, 126. Processingunit 110 of vehicle 200 may be configured to detect a location ofvehicle 200 in each of the plurality of images. Processing unit 110 ofvehicle 200 may also be configured to determine a traveled trajectory3720 based on the detected locations. As used in this disclosure, thetravelled trajectory 3720 may represent an actual path taken by vehicle200 as vehicle 200 travels along road segment 3700.

Processing unit 110 may be configured to determine a current location3712 of vehicle 200 based on analysis of the plurality of images. In oneexemplary embodiment as illustrated in FIG. 37, the current location3712 of vehicle 200 may coincide with a target location 3714 onpredetermined road model trajectory 3710. This may occur, for example,when vehicle 200 is located on predetermined road model trajectory 3710.Although generally vehicle 200 may be expected to be located on or verynear predetermined road model trajectory 3710, it is contemplated thatvehicle 200 may not be located on predetermined road model trajectory3710 as will be discussed below with respect to FIG. 38.

Processing unit 110 may be configured to determine an autonomoussteering action for vehicle 200 by comparing the travelled trajectory3720 with the predetermined road model trajectory 3710 at currentlocation 3712 of vehicle 200. For example, processing unit 110 may beconfigured to determine a transformation (i.e., rotation and potentiallytranslation) such that an error between the travelled trajectory 3720and the predetermined road model trajectory 3710 may be reduced.

Processing unit 110 may be configured to determine a heading direction3730 of vehicle 200 at current location 3712. Processing unit 110 maydetermine heading direction 3730 based on the travelled trajectory 3720.For example, processing unit 110 may determine heading direction 3730 asa gradient of travelled trajectory 3720 at current location 3712 ofvehicle 200. Processing unit 110 may also be configured to determinesteering direction 3740 as a direction tangential to predetermined roadmodel trajectory 3710. In one exemplary embodiment, processing unit 110may be configured to determine steering direction 3740 based on agradient of a three-dimensional polynomial representing predeterminedroad model trajectory 3710.

Processing unit 110 may be configured to determine whether headingdirection 3730 of vehicle 200 is aligned with (i.e., generally parallelto) steering direction 3740 of predetermined road model trajectory 3710.When heading direction 3730 is not aligned with steering direction 3740of predetermined road model trajectory 3710 at current location 3712 ofvehicle 200, processing unit 110 may determine an autonomous steeringaction such that heading direction 3730 of vehicle 200 may be alignedwith steering direction 3740 of predetermined road model trajectory3710. Processing unit 110 may be configured to execute instructionsstored in navigational response module 408 to trigger a desirednavigational response by, for example, turning the steering wheel ofvehicle 200 to achieve a rotation of angle Rotation by the angle mayhelp align heading direction 3730 of vehicle 200 with steering direction3740. Thus, for example, processing unit 110 may perform tail alignmentof vehicle 200 by determining the angle by which vehicle 200 may turn sothat heading direction 3730 of autonomous vehicle may be aligned withsteering direction 3740.

Image acquisition unit 120 may repeatedly acquire the plurality ofimages of the environment in front of vehicle 200, for example, after apredetermined amount of time. Processing unit 110 may also be configuredto repeatedly determine the transformation as discussed above. Thus,image acquisition unit 120 and processing unit 110 may cooperate tonavigate vehicle 200 along road segment 3400 using the travelledtrajectory 3720 (i.e. the “tail”) of vehicle 200.

FIG. 38 illustrates another vehicle 200 travelling on road segment 3700in which disclosed systems and methods for navigating vehicle 200 usingtail alignment. Unlike FIG. 38, vehicle 200 of FIG. 38 is not located onpredetermined road model trajectory 3710. As a result, as illustrated inFIG. 38, target location 3714 of vehicle 200 may not coincide withcurrent location 3712 of vehicle 200.

As discussed above with respect to FIG. 37, processing unit 110 may beconfigured to determine a steering direction 3740 of predetermined roadmodel trajectory 3710 at current location 3712 of vehicle 200.Processing unit 110 may determine steering direction 3740 as thedirection of the gradient of predetermined road model trajectory 3710 attarget location 3714. Processing unit 110 may also be configured todetermine whether heading direction 3730 of vehicle 200 is aligned with(i.e., generally parallel to) steering direction 3740. When headingdirection 3730 is not aligned with steering direction 3740, processingunit 110 may determine a transformation that may include, for example, arotation angle that may be required to align heading direction 3730 withsteering direction 3740. In addition, the transformation may include atranslation “d” that may be required to ensure that vehicle 200 may movefrom current location 3712 to target location 3714 on predetermined roadmodel trajectory 3710.

Processing unit 110 may be configured to determine the transformation bycomparing predetermined road model trajectory 3710 with the travelledtrajectory 3720 of vehicle 200. In one exemplary embodiment, processingunit 110 may determine the transformation by reducing an error betweenpredetermined road model trajectory 3710 and travelled trajectory 3720.Processing unit 110 may be configured to execute instructions stored innavigational response module 408 to trigger a desired navigationalresponse based on the determined transformation.

FIG. 39 is a flowchart showing an exemplary process 3900, for navigatingvehicle 200 along road segment 3700, using tail alignment, consistentwith disclosed embodiments. Steps of process 3900 may be performed byone or more of processing unit 110 and image acquisition unit 120, withor without the need to access memory 140 or 150. The order andarrangement of steps in process 3900 is provided for purposes ofillustration. As will be appreciated from this disclosure, modificationsmay be made to process 3900 by, for example, adding, combining,removing, and/or rearranging the steps for the process.

As illustrated in FIG. 39, process 3900 may include a step 3902 ofacquiring a plurality of images representative of an environment of thevehicle. In one exemplary embodiment, image acquisition unit 120 mayacquire the plurality of images of an area forward of vehicle 200 (or tothe sides or rear of a vehicle, for example) at multiple locations asvehicle travels along road segment 3700. For example, image acquisitionunit 120 may obtain images using image capture device 122 having a fieldof view 202 at each of locations 3752-3768 and 3712 (see FIGS. 37, 38).In other exemplary embodiments, image acquisition unit 120 may acquireimages from one or more of image capture devices 122, 124, 126, havingfields of view 202, 204, 206 at each of locations 3752-3768 and 3712.Image acquisition unit 120 may transmit the one or more images toprocessing unit 110 over a data connection (e.g., digital, wired, USB,wireless, Bluetooth, etc.). Images obtained by the one or more imagecapture devices 122, 124, 126 may be stored in one or more of memories140, 150, and/or database 160.

Process 3900 may also include a step 3904 of determining travelledtrajectory 3720. Processing unit 110 may receive the one or more imagesfrom image acquisition unit 120. Processing unit 110 may executeprocesses similar to those discussed with respect to FIGS. 34-36 toidentify locations 3752-3768 of vehicle 200 in the plurality of images.For example, processing unit 110 may identify one or more landmarks anduse directional vectors of the landmarks to determine locations3752-3768 and current location 3712 using the systems and methodsdisclosed with respect to FIGS. 34-36. Processing unit 110 may determinetravelled trajectory 3720 based on the determined locations 3752-3768and current location 3712 of vehicle 200. In one exemplary embodiment,processing unit 110 may determine travelled trajectory 3720 bycurve-fitting a three-dimensional polynomial to the determined locations3752-3768 and current location 3712 of vehicle 200.

In some embodiments, processing unit 110 may execute monocular imageanalysis module 402 to perform multi-frame analysis on the plurality ofimages. For example, processing unit 110 may estimate camera motionbetween consecutive image frames and calculate the disparities in pixelsbetween the frames to construct a 3D-map of the road. Processing unit110 may then use the 3D-map to detect the road surface as well as togenerate travelled trajectory 3720 of vehicle 200.

Process 3900 may include a step 3906 of determining a current location3712 of vehicle 200. Processing unit 110 may determine current location3712 of vehicle 200 by performing processes similar to those discussed,for example, with respect to FIGS. 34-36 regarding navigation based onrecognized landmarks. In some exemplary embodiments, processing unit 110may determine current location 3712 based on signals from positionsensor 130, for example, a GPS sensor. In another exemplary embodiment,processing unit 110 may determine current location 3712 of vehicle 200by integrating a velocity of vehicle 200 as vehicle 200 travels alongtravelled trajectory 3720. For example, processing unit 110 maydetermine a time “t” required for vehicle 200 to travel between twolocations 3751 and 3712 on travelled trajectory 3720. Processing unit110 may integrate the velocity of vehicle 200 over time t to determinecurrent location 3712 of vehicle 200 relative to location 3751.

Process 3900 may also include a step 3908 of determining whether currentlocation 3712 of vehicle 200 is located on predetermined road modeltrajectory 3710. In some exemplary embodiments, predetermined road modeltrajectory 3710 may be represented by a three-dimensional polynomial ofa target trajectory along road segment 3700. Processing unit 110 mayretrieve predetermined road model trajectory 3710 from database 160stored in one or memories 140 and 150 included in vehicle 200. In someembodiments, processing unit 110 may retrieve predetermined road modeltrajectory 3710 from database 160 stored at a remote location via awireless communications interface.

Processing unit 110 may determine whether current location 3712 ofvehicle 200 is located on predetermined road model trajectory 3710,using processes similar to those discussed with respect to FIGS. 34-37,by for example, determining a distance between vehicle 200 and arecognized landmark. When processing unit 110 determines that currentlocation of vehicle 200 is on predetermined road model trajectory 3710(see FIG. 37), processing unit 110 may proceed to step 3912. Whenprocessing unit 110 determines, however, that current location ofvehicle 200 is not on predetermined road model trajectory 3710 (see FIG.38), processing unit 110 may proceed to step 3910.

In step 3910, processing unit 110 may determine a lateral offset “d”that may help ensure that vehicle 200 may move from current location3712 to target location 3714 on predetermined road model trajectory3710. Processing unit 110 may determine lateral offset d. In oneembodiment, processing unit 110 may determine lateral offset d bydetermining the left and right sides 3706, 3708. In one other exemplaryembodiments, processing unit 110 may determine a translation functionneeded to convert current location 3712 to target location 3714. Inanother embodiment, processing unit 110 may determine the translationfunction by reducing the error between current location 3712 and targetlocation 3714. In additional exemplary embodiments, processing unit 110may determine the lateral offset d by observing (using one or moreonboard cameras and one or more images captured by those cameras) leftside 3706 and right side 3708 of road segment 3700. After determiningthe lateral offset d, processing unit may proceed to step 3912.

Process 3900 may include a step 3912 of determining heading direction3730 of vehicle 200, and possibly a correction to the current location3712 computed in step 3906. In one exemplary embodiment, processing unit110 may determine heading direction 3730 and a correction to location3712 by aligning the travelled trajectory 3720 at current location 3712with the model trajectory 3710. The alignment procedure may provide arigid transformation that reduces or minimizes the distance between 3720and 3712. In one exemplary embodiment, processing unit 110 may compute arigid transformation with four degrees of freedom, accounting for 3Drotation (heading) and 1D longitudinal translation. In another exemplaryembodiment, processing unit 110 may compute a rigid transformation withany number of parameters (degrees of freedom) between 1 and 6. Afteralignment, processing unit 110 may determine the predicted location 3774(see FIGS. 37, 38) of vehicle 200 after time “t” based on a currentvelocity of vehicle 200 and the geometry of the model trajectory 3710.

In other exemplary embodiments, in step 3912, processing unit 110 maydetermine heading direction 3730 of vehicle 200, and possibly acorrection to the current location 3712 computed in step 3906. Forexample, processing unit 110 may determine heading direction 3730 andimproved location 3712 by aligning the travelled trajectory 3720 atcurrent location 3712 with the model trajectory 3710. The alignmentprocedure may find a rigid transformation that minimizes the distancebetween 3720 and 3712. In one exemplary embodiment, processing unit 110may compute a rigid transformation with four degrees of freedom,accounting for 3D rotation (heading) and 1D longitudinal translation. Inanother exemplary embodiment, processing unit 110 may compute a rigidtransformation with any number of parameters (degreed of freedom)between 1 and 6. After alignment, processing unit 110 may determine thepredicted location 3774 (see FIGS. 37, 38) of vehicle 200 after time “t”based on a current velocity of vehicle 200 and the geometry of the modeltrajectory 3710.

In yet other exemplary embodiments, processing unit 110 may determineheading direction 3730 and a location 3712 as a gradient of travelledtrajectory 3720 at current location 3712 of vehicle 200. For example,processing unit 110 may obtain a slope of a three-dimensional polynomialrepresenting travelled trajectory 3720 to determine heading direction3730 of vehicle 200. In another exemplary embodiment, process 110 mayproject travelled trajectory 3720 forward from current location 3712. Inprojecting travelled trajectory 3720, processing unit 110 may determinea predicted location 3774 (see FIGS. 37, 38) of vehicle 200 after time“t” based on a current velocity of vehicle 200.

Processing unit 110 may also determine predicted location 3774 ofvehicle 200 after time “t” based on one of many cues. For example,processing unit 110 may determine predicted location 3774 of vehicle 200after time “t” based on a left lane mark polynomial, which may be apolynomial representing left side 3706 of road segment 3700. Thus, forexample, processing unit 110 may determine left position 3770 (see FIGS.37, 38) on the left lane mark polynomial corresponding to currentlocation 3712 of vehicle 200. Processing unit 110 may determine location3770 by deter raining the distance “D” between current location 3712 andleft side 3706 based on the left lane mark polynomial. It iscontemplated that when vehicle 200 is not located on predetermined roadmodel trajectory 3710 (as in FIG. 38), processing unit 110 may determinedistance D as the distance between target location 3714 and left side3706. Processing unit 110 may also determine a location 3772 on leftside 3706 after time “t” using the mathematical representation of theleft lane mark polynomial and current velocity of vehicle 200.Processing unit 110 may determine predicted location 3774 of vehicle 200by laterally offsetting the determined location 3773 on left side 3706by distance D. In another exemplary embodiment, processing unit 110 maydetermine the location of vehicle 200 after time “t” based on a rightlane mark polynomial, which may be a polynomial representing right side3708 of road segment 3700. Processing unit 110 may perform processessimilar to those discussed above with respect to left lane markpolynomial to determine predicted position 3774 of vehicle 200 based onright lane mark polynomial.

In some exemplary embodiments, processor 110 may determine the locationof vehicle 200 after time “t” based on the trajectory followed by aforward vehicle, which may be travelling in front of vehicle 200. Inother exemplary embodiments, processing unit 200 may determine thelocation of vehicle 200 after time “t” by determining an amount of freespace ahead of vehicle 200 and a current velocity of vehicle 200. Insome embodiments, processing unit 200 may determine the location ofvehicle 200 after time “t” based on virtual lanes or virtual laneconstraints. For example, when processing unit 110 detects two vehiclestravelling in front of vehicle 200, one in each adjacent lane,processing unit 110 may use the average lateral distance between the twovehicles in front as a trajectory (virtual lane marker), which may beused to determine a position of vehicle 200 after time “t.” In otherembodiments, processing unit 110 may use mathematical representations ofleft side 3706 (i.e. left lane mark polynomial) and right side 3708(i.e. right lane mark polynomial) as defining virtual lane constraints.Processing unit 110 may determine predicted position 3774 of vehicle 200based on the virtual lane constraints (i.e. based on both the left andthe right lane mark polynomials) and an estimated location of vehicle200 from the left and right sides 3706, 3708.

In other embodiments, processing unit 110 may determine predictedlocation 3774 of vehicle 200 after time “t” based on following atrajectory predicted using holistic path prediction methods. In someexemplary embodiments, processing unit 110 may determine predictedlocation 3774 of vehicle 200 after time “t” by applying weights to someor all of the above-described cues. For example, processing unit 110 maydetermine the location of vehicle 200 after time “t” as a weightedcombination of the locations predicted based on one or more of a leftlane mark polynomial model, a right lane mark polynomial model, holisticpath prediction, motion of a forward vehicle, determined free spaceahead of the autonomous vehicle, and virtual lanes. Processing unit 110may use current location 3712 of vehicle 200 and predicted location 3774after time “t” to determine heading direction 3730 for vehicle 200.

In some embodiments, in step 3912 of process 3900, processing unit 110may also estimate a longitudinal offset. For example, processing unit110 may solve for the heading and the offset by an alignment procedure,between the model trajectory and the tail of vehicle 200.

Process 3900 may also include a step 3914 of determining steeringdirection 3740. In one exemplary embodiment, processing unit 110 mayobtain a mathematical representation (e.g. three-dimensional polynomial)of predetermined road model trajectory 3710. Processing unit 110 maydetermine steering direction 3740 as a vector oriented tangentially topredetermined road model trajectory 3710 at target location 3714. Forexample, processing unit 110 may determine direction 3740 as a vectorpointing along a gradient of the mathematical representation ofpredetermined road model trajectory 3710 at target location 3714.

Process 3900 may also include a step 3916 of adjusting steering system240 of vehicle 200 based on the transformation determined, for example,in steps 3910-3914. The required transformation may include lateraloffset d. The transformation may further include rotation by an angle tohelp ensure that heading direction 3730 of vehicle 200 may be alignedwith steering direction 3740. Although, FIGS. 37, 38 illustratedetermination of one angle between heading direction 3730 and steeringdirection 3740, it is contemplated that in three-dimensional space,rotation along three angles in three generally orthogonal planes may berequired to ensure that heading direction 3730 may be aligned withsteering direction 3730. One of ordinary skill in the art would,therefore, recognize that the transformation determined in steps3910-3914 may include at least three rotational angles and at least onetranslation (i.e. lateral offset).

Processing unit 110 may send control signals to steering system 240 toadjust rotation of the wheels of vehicle 200 so that heading direction3730 may be aligned with steering direction 3740 and vehicle 200 maymove from current location 3712 to target location 3714 when vehicle 200is located off predetermined road model trajectory 3710. Processing unit110 and/or image acquisition unit 120 may repeat steps 3902 through 3916after a predetermined amount of time. In one exemplary embodiment, thepredetermined amount of time may range between about 0.5 seconds to 1.5seconds. By repeatedly determining lateral offset d and rotation angles,processing unit 110 and/or image acquisition unit 120 may help tonavigate vehicle 200, using tail alignment, along road segment 3700.

As discussed in other sections, navigation of an autonomous vehiclealong a road segment may include the use of one or more recognizedlandmarks. Among other things, such recognized landmarks may enable theautonomous vehicle to determine its current location with respect to atarget trajectory from sparse data model 800. The current locationdetermination using one or more recognized landmarks may be more precisethan determining a position using GPS sensing, for example.

Between recognized landmarks, the autonomous vehicle may navigate usinga dead-reckoning technique. This technique may involve periodicallyestimating a current location of the vehicle with respect to the targettrajectory based on sensed ego-motion of the vehicle. Such sensed egomotion may enable the vehicle (e.g., using processing unit 110) to notonly estimate the current location of the vehicle relative to the targettrajectory, but it may also enable the processing unit 110 toreconstruct the vehicle's travelled trajectory. Sensors that may be usedto determine the ego motion of the vehicle may include various sensorssuch as, for example, onboard cameras, speedometers, and/oraccelerometers. Using such sensors, processing unit 110 may sense wherethe vehicle has been and reconstruct the travelled trajectory. Thisreconstructed travelled trajectory may then be compared to the targettrajectory using the tail alignment technique described above todetermine what navigational changes, if any, are required to align thetraveled trajectory at a current location with the target trajectory atthe current location.

Navigating Road Junctions

Consistent with disclosed embodiments, the system may navigate throughroad junctions, which may constitute areas with few or no lane markings.Junction navigation may include 3D localization based on two or morelandmarks. Thus, for example, the system may rely on two or morelandmarks to determine a current location and a heading of an autonomousvehicle. Further, the system may determine a steering action based onthe determined heading and a direction of a predetermine road modeltrajectory representing a preferred path for the vehicle.

FIG. 40 illustrates vehicle 200 (which may be an autonomous vehicle)travelling through road junction 4000 in which the disclosed systems andmethods for navigating road junctions may be used. As illustrated inFIG. 40, vehicle 200 may be travelling along road segment 4002, whichmay intersect with road segment 4004. Although road segments 4002 and4004 appear to intersect at right angles in FIG. 40, it is contemplatedthat road segments 4002 and 4004 may intersect at any angle. Further,although road segments 4002 and 4004 each have two lanes in FIG. 40, itis contemplated that road segments 4002 and 4004 may have any number oflanes. It is also contemplated that road segments 4002 and 4004 may havethe same number or different number of lanes.

Vehicle 200 may travel along lane 4006 of road segment 4002. Vehicle 200may be equipped with three image capture devices 122, 124, 126.Although, FIG. 40 depicts vehicle 200 as equipped with image capturedevices 122, 124, 126, more or fewer image capture devices may beemployed on any particular vehicle 200. As illustrated in FIG. 40, lane4006 of road segment 4002 may be delimited by left side 4008 and rightside 4010. A predetermined road model trajectory 4012 may define apreferred path (i.e., a target road model trajectory) within lane 4006of road segments 4002, 4004 that vehicle 200 may follow as vehicle 200travels along road segments 4002, 4004 through junction 4000. In someexemplary embodiments, predetermined road model trajectory 4012 may belocated equidistant from left side 4008 and right side 4010. It iscontemplated however that predetermined road model trajectory 4012 maybe located nearer to one or the other of left side 4008 and right side4010 of road segment 4002.

In one exemplary embodiment, predetermined road model trajectory 4012may be mathematically defined using a three-dimensional polynomialfunction. In some exemplary embodiments, processing unit 110 of vehicle200 may be configured to retrieve predetermined road model trajectory4012 from a database (e.g. 160) stored in one or more of memories 140,150 included in vehicle 200. In other exemplary embodiments, processingunit 110 of vehicle 200 may be configured to retrieve predetermined roadmodel trajectory 4012 from a database (e.g. 160), which may be storedremotely from vehicle 200, over a wireless communications interface. Asillustrated in the exemplary embodiment of FIG. 40, predetermined roadmodel trajectory 4012 may allow vehicle 200 to turn left from lane 4006of road segment 4002 into lane 4014 of road segment 4004.

Image acquisition unit 120 may be configured to acquire an imagerepresentative of an environment of vehicle 200. For example, imageacquisition unit 120 may obtain an image showing a view in front ofvehicle 200 using one or more of image capture devices 122, 124, 126.Processing unit 110 of vehicle 200 may be configured to detect two ormore landmarks 4016, 4018 in the one or more images acquired by imageacquisition unit 120. Such detection may occur using the landmarkdetection techniques previously discussed, for example. Processing unit110 may detect the two or more landmarks 4016, 4018 using one or moreprocesses of landmark identification discussed above with reference toFIGS. 22-28. Although FIG. 40 illustrates two landmarks 4016, 4018, itis contemplated that vehicle 200 may detect more than two landmarks4016, 4018 (i.e., three or more landmarks) based on the images acquiredby image acquisition unit 120. For example, FIG. 40 illustratesadditional landmarks 4020 and 4022, which may be detected and used byprocessing unit 110.

Processing unit 110 may be configured to determine positions 4024, 4026of landmarks 4016, 4018, respectively, relative to vehicle 200.Processing unit 110 may also be configured to determine one or moredirectional indicators 4030, 4032 of landmarks 4016, 4018 relative tovehicle 200. Further, processing unit 110 may be configured to determinecurrent location 4028 of vehicle 200 based on an intersection ofdirectional indicators 4030, 4032. In one exemplary embodiment asillustrated in FIG. 40, processing unit 110 may be configured todetermine current location 4028 as the intersection point of directionalindicators 4030, 4032.

Processing unit 110 may be configured to determine previous location4034 of vehicle 200. In one exemplary embodiment, processing unit 110may repeatedly determine a location of vehicle 200 as vehicle 200travels on road segments 4002 and 4004. Thus, for example, beforevehicle 200 reaches its current location 4028, vehicle may be located atprevious location 4034 and may travel from previous location 4034 tocurrent location 4028. Before reaching current location 4028, processingunit 110 of vehicle 200 may be configured to determine positions 4024,4026 of landmarks 4016, 4018, respectively, relative to vehicle 200.Processing unit 110 may also be configured to determine directionalindicators 4036, 4038 of landmarks 4016, 4018 relative to vehicle 200.Processing unit 110 may also be configured to determine previouslocation 4034 of vehicle 200 based on an intersection of directionalindicators 4036, 4038. In one exemplary embodiment as illustrated inFIG. 40, processing unit 110 may be configured to determine previouslocation 4034 as the intersection point of directional indicators 4036,4038.

Processing unit 110 may be configured to determine a direction 4040 ofpredetermined road model trajectory 4012 at current location 4028 ofvehicle 200. Processing unit 110 may determine direction 4040 as adirection tangential to predetermined road model trajectory 4012. In oneexemplary embodiment, processing unit 110 may be configured to determinedirection 4040 based on a gradient or slope of a three-dimensionalpolynomial representing predetermined road model trajectory 4012.

Processing unit 110 may also be configured to determine headingdirection 4050 of vehicle 200. Processing unit 110 may determine headingdirection 4050 based on landmarks 4016 and 4018. Processing unit 110 maydetermine heading direction 4050 based on current location 4028 andprevious location 4034 of vehicle 200. For example, processing unit 110may determine heading direction 4050 as a vector extending from previouslocation 4034 towards current location 4028. In some exemplaryembodiments, processing unit 110 may determine heading direction 4050 asa direction along which image capture device 122 may be orientedrelative to a local coordinate system associated with vehicle 200.

Processing unit 110 may be configured to determine whether headingdirection 4050 of vehicle 200 is aligned with (i.e., generally parallelto) direction 4040 of predetermined road model trajectory 4012. Whenheading direction 4050 is not aligned with direction 4040 ofpredetermined road model trajectory 4012 at current location 4028 ofvehicle 200, processing unit 110 may determine a steering angle betweenheading direction 4050 of vehicle 200 and direction 4040 ofpredetermined road model trajectory 4012. In one exemplary embodiment,processing unit 110 may also determine, for example, a reduction oracceleration in a current velocity of vehicle 200 required to helpensure that heading direction 4050 of vehicle 200 may be aligned withdirection 4040 of predetermined road model trajectory 4012 in apredetermined amount of time. Processing unit 110 may be configured toexecute instructions stored in navigational response module 408, forexample, to transmit a control signal specifying the steering angle tosteering system 240 of the vehicle. Steering system 200, in turn, may beconfigured to rotate wheels of vehicle 200 to help ensure that headingdirection 4050 of vehicle 200 may be aligned with direction 4040 ofpredetermined road model trajectory 4012.

Image acquisition unit 120 may repeatedly acquire an image of theenvironment in front of vehicle 200, for example, after a predeterminedamount of time. Processing unit 110 may also be configured to repeatedlydetect landmarks 4016, 4018, 4020, 4022, etc., in the image acquired byimage acquisition unit 120 and determine the steering angle as discussedabove. Thus, image acquisition unit 120 and processing unit 110 maycooperate to navigate vehicle 200 through junction 400 using two or moreof landmarks 4016, 4018, 4020, 4022.

FIG. 41 is a flowchart showing an exemplary process 4100, for navigatingvehicle 200 through junction 4000, using two or more landmarks 4016,4018, 4020, 4022, consistent with disclosed embodiments. Steps ofprocess 4100 may be performed by one or more of processing unit 110 andimage acquisition unit 120, with or without the need to access memory140 or 150. The order and arrangement of steps in process 4100 isprovided for purposes of illustration. As will be appreciated from thisdisclosure, modifications may be made to process 4100 by, for example,adding, combining, removing, and/or rearranging the steps for theprocess.

As illustrated in FIG. 41, process 4100 may include a step 4102 ofacquiring an image representative of an environment of the vehicle. Inone exemplary embodiment, image acquisition unit 120 may acquire one ormore images of an area forward of vehicle 200 (or to the sides or rearof a vehicle, for example). For example, image acquisition unit 120 mayobtain an image using image capture device 122 having a field of view202. In other exemplary embodiments, image acquisition unit 120 mayacquire images from one or more of image capture devices 122, 124, 126,having fields of view 202, 204, 206. Image acquisition unit 120 maytransmit the one or more images to processing unit 110 over a dataconnection (e.g., digital, wired, USB, wireless, Bluetooth, etc.).

Process 4100 may also include a step 4104 of identifying two or morelandmarks 4016, 4018, 4020, 4022 in the one or more images. Processingunit 110 may receive the one or more images from image acquisition unit120. Processing unit 110 may execute monocular image analysis module 402to analyze the plurality of images at step 4104, as described in furtherdetail in connection with FIGS. 5B-5D. By performing the analysis,processing unit 110 may detect a set of features within the set ofimages, for example, two or more landmarks 4016, 4018, 4020, 4022.Landmarks 4016, 4018, 4020, 4022 may include one or more traffic signs,arrow markings, lane markings, dashed lane markings, traffic lights,stop lines, directional signs, reflectors, landmark beacons, lampposts,a change in spacing of lines on the road, signs for businesses, and thelike.

In some embodiments, processing unit 110 may execute monocular imageanalysis module 402 to perform multi-frame analysis on the plurality ofimages to detect two or more landmarks 4016, 4018, 4020, 4022. Forexample, processing unit 110 may estimate camera motion betweenconsecutive image frames and calculate the disparities in pixels betweenthe frames to construct a 3D-map of the road. Processing unit 110 maythen use the 3D-map to detect the road surface, as well as landmarks4016, 4018, 4020, 4022. In another exemplary embodiment, image processor190 of processing unit 110 may combine a plurality of images receivedfrom image acquisition unit 120 into one or more composite images.Processing unit 110 may use the composite images to detect the two ormore landmarks 4016, 4018, 4020, 4022. For example, in some embodiments,processing unit 110 may perform stereo processing of images from two ormore image capture devices.

Process 4100 may also include a step 4106 of determining directionalindicators 4030, 4032 associated with at least two landmarks 4016, 4018,respectively. Processing unit 110 may determine directional indicators4030, 4032 based on the positions 4024, 4026 of the at least twolandmarks 4016, 4018, respectively, relative to vehicle 200. Forexample, processing unit 110 may receive landmark positions 4024, 4026for landmarks 4016, 4018, respectively, from information, which may bestored in one or more databases in memory 140 or 150. Processing unit110 may determine directional indicator 4030 as a vector extending fromvehicle 200 towards landmark position 4024. Likewise, processing unit110 may determine directional indicator 4032 as a vector extending fromvehicle 200 towards landmark position 4026. Although two landmarks 4016,4018 are referenced in the above discussion, it is contemplated thatprocessing unit 110 may determine landmark positions 4024, 4026, anddirectional indicators 4030, 4032 for more than two landmarks 4016, 4018(e.g., for landmarks 4020, 4022).

Process 4100 may include a step 4108 of determining current location4028 of vehicle 200. Processing unit 110 may determine current location4028 based on an intersection of directional indicators 4030 and 4032 oflandmarks 4016, 4018, respectively (e.g., at an intersection point ofdirectional indicators 4030 and 4032). Process 4100 may include a step4110 of determining previous location 4034 of vehicle 200. As discussedabove, processing unit 110 may be configured to determine previouslocation 4034 of vehicle 200 based on two or more landmarks 4016, 4018,4020, 4022. In one exemplary embodiment, processing unit 110 mayrepeatedly determine a location of vehicle 200 using two or morelandmarks 4016, 4018, 4020, 4022 as vehicle 200 moves on road segments4002 and 4004. Thus, for example, before vehicle 200 reaches its currentlocation 4028, vehicle may be located at previous location 4034 and maytravel from previous location 4034 to current location 4028. Beforereaching current location 4028, processing unit 110 of vehicle 200 maybe configured to determine positions 4024, 4026 of landmarks 4016, 4018,respectively, relative to vehicle 200. Processing unit 110 may performprocesses similar to those discussed above with respect to step 4108 todetermine previous location 4034 of vehicle 200. For example, processingunit 110 may be configured to determine directional indicators 4036,4038 of landmarks 4016, 4018 relative to vehicle 200. Processing unit110 may also be configured to determine previous location 4034 ofvehicle 200 based on an intersection of directional indicators 4036,4038 (e.g. at an intersection point of directional indicators 4036 and4038).

Process 4100 may include a step 4112 of determining heading direction4050 of vehicle 200. As discussed above, processing unit 110 maydetermine heading direction 4050 based on current location 4028 andprevious location 4034 of vehicle 200, both of which may be determinedusing two or more of landmarks 4016, 4018, 4020, 4022. In one exemplaryembodiment, processing unit 110 may determine heading direction 4050 asa vector extending from previous location 4034 towards current location4028. In another exemplary embodiment, processing unit 110 may determineheading direction 4050 as a direction along which image capture device122 may be oriented relative to a local coordinate system associatedwith vehicle 200. Although only two landmarks 4016, 4018 have beendescribed with respect to determining current location 4028 and previouslocation 4024 of vehicle 200, it is contemplated that processing unitmay use more than two landmarks 4016, 4018 to determine current location4028 and previous location 4024 of vehicle 200 and heading direction4050.

Process 4100 may include a step 4114 of determining direction 4040 ofpredetermined road model trajectory 4012 at current location 4028 ofvehicle 200. In one exemplary embodiment, processing unit 110 may obtaina mathematical representation (e.g. three-dimensional polynomial) ofpredetermined road model trajectory 4012. Processing unit 110 maydetermine direction 4040 as a vector oriented tangentially topredetermined road model trajectory 4012 at current location 4028 ofvehicle 200. For example, processing unit 110 may determine direction4040 as a vector pointing along a gradient of the mathematicalrepresentation of predetermined road model trajectory 4012 at currentlocation 4028 of vehicle 200. Although the above description assumesthat current location 4028 and previous location 4034 of vehicle 200 arelocated on predetermined road model trajectory 4012, processing unit 110may perform processes similar to those discussed above with respect toFIGS. 34-39 when vehicle 200 is not located on predetermined road modeltrajectory 4012. For example, processing unit 110 may determine atransform required to move vehicle 200 to predetermined road modeltrajectory 4012 before determining direction 4040 as discussed above.

Process 4100 may also include a step 4116 of determining steering angle□ for vehicle 200. Processing unit 110 may also determine steering angle□ as an angle between heading direction 4050 and direction 4040 ofpredetermined road model trajectory 4012 at current location 4028 ofvehicle 200. Processing unit 110 may execute instructions innavigational module 408, for example, to transmit a control signalspecifying steering angle □ to steering system 240. Steering system 240may help adjust, for example, a steering wheel of vehicle 200 to turnthe wheels of vehicle 200 to help ensure that heading direction 4050 ofvehicle 200 may be aligned (i.e., parallel) with direction 4040 ofpredetermined road model trajectory 4012.

Processing unit 110 and/or image acquisition unit 120 may repeat steps4102 through 4116 after a predetermined amount of time. In one exemplaryembodiment, the predetermined amount of time may range between about 0.5seconds to 1.5 seconds. By repeatedly determining current location 4028,heading direction 4050, direction 4040 of predetermined road modeltrajectory 4012 at current location 4028, and steering angle required toalign heading direction 4050 with direction 4040, processing unit 110may transmit one or more control signals to one or more of throttlingsystem 220, steering system 240, and braking system 230 to navigatevehicle 200 through road junction 4000, using two or more landmarks4016, 4018, 4020, 4022.

Navigation Using Local Overlapping Maps

Consistent with disclosed embodiments, the system may use a plurality oflocal maps for navigation. Each map may have its own arbitrarycoordinate frame. To ease the transition in navigating from one localmap to another, the maps may include an overlap segment, and navigationin the overlap segment may be based on both of the overlapping maps.

FIG. 43 illustrates first and second local maps 4200 and 4202 associatedwith first and second road segments 4204 and 4206, respectively. Firstroad segment 4204 may be different from second road segment 4206. Maps4200 and 4202 may each have their own arbitrary coordinate frame. Maps4200 and 4202 may also each constitute a sparse map having the same ordifferent data densities. In one exemplary embodiment, maps 4200 and4202 may each have a data density of no more than 10 kilobytes perkilometer. Of course, local maps 4200 and 4202 may include other datadensity values, such as any of the data densities previously discussedrelative to sparse map 800, for example. Vehicle 200 (which may be anautonomous vehicle) travelling on a first road segment 4204 and/or onsecond road segment 4206 may use the disclosed systems and methods fornavigation. Vehicle 200 may include at least one image capture device122, which may be configured to obtain one or more images representativeof an environment of the autonomous vehicle. Although FIG. 42 depictsvehicle 200 as equipped with image capture devices 122, 124, 126, moreor fewer image capture devices may be employed on any particular vehicle200. As illustrated in FIG. 42, map 4200 may include road segment 4204,which may be delimited by left side 4208 and right side 4210. Apredetermined road model trajectory 4212 may define a preferred path(i.e., a target road model trajectory) within road segment 4204.Predetermined road model trajectory 4212 may be mathematicallyrepresented by a three-dimensional polynomial. Vehicle 200 may followpredetermined road model trajectory 4212 as vehicle 200 travels alongroad segment 4204. In some exemplary embodiments, predetermined roadmodel trajectory 4212 may be located equidistant from left side 4208 andright side 4210. It is contemplated however that predetermined roadmodel trajectory 4212 may be located nearer to one or the other of leftside 4208 and right side 4210 of road segment 4204. As also illustratedin FIG. 42, a portion of road segment 4204 between delimiting points Aand B may represent an overlap segment 4220. As will be described later,overlap segment 4220 between positions A and B of road segment 4204 mayoverlap with a portion of road segment 4206. Further, although FIG. 42illustrates one lane in road segment 4204, it is contemplated that roadsegment 4204 may have any number of lanes. It is also contemplated thatvehicle 200 travelling along any lane of road segment 4204 may benavigated according to the disclosed methods and systems. Further, insome embodiments, a road segment may extend between two known locationssuch as, for example, two intersections.

As also illustrated in FIG. 42, map 4202 may include road segment 4206,which may be delimited by left side 4222 and right side 4224. Apredetermined road model trajectory 4226 may define a preferred path(i.e., a target road model trajectory) within road segment 4206.Predetermined road model trajectory 4226 may be mathematicallyrepresented by a three-dimensional polynomial. Vehicle 200 may followpredetermined road model trajectory 4226 as vehicle 200 travels alongroad segment 4206. In some exemplary embodiments, predetermined roadmodel trajectory 4226 may be located equidistant from left side 4222 andright side 4224. It is contemplated however that predetermined roadmodel trajectory 4226 may be located nearer to one or the other of leftside 4222 and right side 4224 of road segment 4206. As also illustratedin FIG. 42, a portion of road segment 4206 between delimiting points A′and B′ may represent overlap segment 4220, which may overlap withoverlap segment 4220 between delimiting points A and B of road segment4204. Although FIG. 42 illustrates one lane in road segment 4206, it iscontemplated that road segment 4206 may have any number of lanes. It isalso contemplated that vehicle 200 travelling along any lane of roadsegment 4206 may be navigated according to the disclosed methods andsystems.

As used in this disclosure, the term overlap indicates that overlapsegment 4220 represents the same portion of the road that may betravelled on by vehicle 200. In some embodiments, an overlap segment4220 may include a segment of map 4200 that represents a road segmentand associated road features (such as landmarks, etc.) that are alsorepresented by a corresponding segment (i.e., the overlap segment) ofmap 4222. As a result, overlap segment 4220 may include portions of roadsegments 4204, 4206 having the same size (length, width, height, etc.),shapes (orientation and inclination, etc.), etc. Moreover, the shapesand lengths of predetermined road model trajectories 4212 and 4226 inthe overlap segment 4220 may be similar. However, because maps 4200 and4202 may have different local coordinate systems, the mathematicalrepresentations (e.g., three-dimensional polynomials) of predeterminedroad model trajectories 4212 and 4226 may differ in the overlap segment4220. In one exemplary embodiment, overlap segment 4220 may have alength ranging between 50 m and 150 m.

Image acquisition unit 120 may be configured to acquire an imagerepresentative of an environment of vehicle 200. For example, imageacquisition unit 120 may obtain an image showing a view in front ofvehicle 200 using one or more of image capture devices 122, 124, 126.Processing unit 110 of vehicle 200 may be configured to detect a currentlocation 4214 of vehicle 200 using one or more navigational processesdiscussed above with reference to FIGS. 34-36. Processing unit 110 mayalso be configured to determine whether current position 4214 of vehicle200 lies on road segment 4204 or 4206 using one or more processes ofdetermining intersection points of directional vectors for recognizedlandmarks with one or more of predetermined road model trajectories 4212and 4226 as discussed above with reference to FIGS. 34-36. Furthermore,processing unit 110 may be configured to determine whether currentlocation 4214 of vehicle 200 lies on road segment 4204, road segment4206, or in the overlap segment 4220, using similar processes discussedabove with reference to FIGS. 34-36.

When vehicle 200 is located on road segment 4204, processing unit 110may be configured to align a local coordinate system of vehicle 200 witha local coordinate system associated with road segment 4204. Afteraligning the two coordinate systems, processing unit 110 may beconfigured to determine a direction 4230 of predetermined road modeltrajectory 4212 at current location 4214 of vehicle 200. Processing unit110 may determine direction 4230 as a direction tangential topredetermined road model trajectory 4212. In one exemplary embodiment,processing unit 110 may be configured to determine direction 4230 basedon a gradient or slope of a three-dimensional polynomial representingpredetermined road model trajectory 4212.

Processing unit 110 may also be configured to determine headingdirection 4240 of vehicle 200. As illustrated in FIG. 42, headingdirection 4240 of vehicle 200 may be a direction along which imagecapture device 122 may be oriented relative to the local coordinatesystem associated with vehicle 200. Processing unit 110 may beconfigured to determine whether heading direction 4240 of vehicle 200 isaligned with (i.e., generally parallel to) direction 4230 ofpredetermined road model trajectory 4212. When heading direction 4240 isnot aligned with direction 4230 of predetermined road model trajectory4212 at current location 4214 of vehicle 200, processing unit 110 maydetermine a first autonomous navigational response (ANR) that may helpensure that heading direction 4240 of vehicle 200 may be aligned withdirection 4230 of predetermined road model trajectory 4212.

In one exemplary embodiment, first ANR may include, for example, adetermination of an angle by which the steering wheel or front wheels ofvehicle 200 may be turned to help ensure that heading direction 4240 ofvehicle 200 may be aligned with direction 4230 of predetermined roadmodel trajectory 4212. In another exemplary embodiment, first autonomousnavigational response may also include a reduction or acceleration in acurrent velocity of vehicle 200 to help ensure that heading direction4240 of vehicle 200 may be aligned with direction 4230 of predeterminedroad model trajectory 4212 in a predetermined amount of time. Processingunit 110 may be configured to execute instructions stored innavigational response module 408 to trigger first ANR by, for example,turning the steering wheel of vehicle 200 to achieve a rotation of anangle 1. Rotation by angle 1 may help align heading direction 4240 ofvehicle 200 with direction 4230.

When vehicle 200 is located on road segment 4206, processing unit 110may be configured to align a local coordinate system of vehicle 200 witha local coordinate system associated with road segment 4206. Afteraligning the two coordinate systems, processing unit 110 may beconfigured to determine a direction 4250 of predetermined road modeltrajectory 4226 at current location 4214 of vehicle 200. Processing unit110 may determine direction 4250 as a direction tangential topredetermined road model trajectory 4226. In one exemplary embodiment,processing unit 110 may be configured to determine direction 4250 basedon a gradient or slope of a three-dimensional polynomial representingpredetermined road model trajectory 4226.

Processing unit 110 may also be configured to determine headingdirection 4260 of vehicle 200. As illustrated in FIG. 42, headingdirection 4260 of vehicle 200 may be a direction along which imagecapture device 122 may be oriented relative to the local coordinatesystem associated with vehicle 200. Processing unit 110 may beconfigured to determine whether heading direction 4260 of vehicle 200 isaligned with (i.e., generally parallel to) direction 4250 ofpredetermined road model trajectory 4226. When heading direction 4260 isnot aligned with direction 4250 of predetermined road model trajectory4226 at current location 4214 of vehicle 200, processing unit 110 maydetermine a second ANR that may help ensure that heading direction 4260of vehicle 200 may be aligned with direction 4250 of predetermined roadmodel trajectory 4226.

In one exemplary embodiment, the second ANR may include, for example, adetermination of an angle 2 by which the steering wheel or front wheelsof vehicle 200 may be turned to help ensure that heading direction 4260of vehicle 200 may be aligned with direction 4250 of predetermined roadmodel trajectory 4226. In another exemplary embodiment, the second ANRmay also include a reduction or acceleration in a current velocity ofvehicle 200 to help ensure that heading direction 4260 of vehicle 200may be aligned with direction 4250 of predetermined road modeltrajectory 4226 in a predetermined amount of time. Processing unit 110may be configured to execute instructions stored in navigationalresponse module 408 to trigger second ANR by, for example, turning thesteering wheel of vehicle 200 to achieve a rotation of angle 2. Rotationby angle 2 may help align heading direction 4260 of vehicle 200 withdirection 4250.

When vehicle 200 is located on overlap segment 4220 of road segments4204, 4206, processing unit 110 may be configured to align the localcoordinate system of vehicle 200 with both the local coordinate systemassociated with road segment 4204 as well as the local coordinate systemassociated with road segment 4206. Thus, processing unit 110 may beconfigured to determine a third ANR based on both maps 4200 and 4202. Inone exemplary embodiment, processing unit 110 may determine the thirdANR as an angle 3 by which the steering wheel or front wheels of vehicle200 may be turned to help ensure that heading direction 4240 of vehicle200 may be aligned with heading direction 4230 of predetermined roadmodel trajectory, and heading direction 4260 of vehicle 200 may bealigned with direction 4250 of predetermined road model trajectory 4226.Thus, for example, processing unit 110 may determine angle □₃ as acombination of angles 1 and 2.

Image acquisition unit 120 may repeatedly acquire an image of theenvironment in front of vehicle 200, for example, after a predeterminedamount of time. Processing unit 110 may also be configured to repeatedlydetect whether current location 4214 of vehicle 200 lies on road segment4204, road segment 4206, or in overlap segment 4220. Processing unit 110may determine first, second, or third ANR (e.g., angles 1, 2, or 3)based on where vehicle 200 is located on road segments 4204, 4206. Thus,image acquisition unit 120 and processing unit 110 may cooperate tonavigate vehicle 200 along road segments 4204 and 4206 using overlapsegment 4220.

FIGS. 43A-C include flowcharts showing an exemplary process 4300, fornavigating vehicle 200 along road segments 4204, 4206, using overlappingmaps 4200, 4202, consistent with disclosed embodiments. Steps of process4300 may be performed by one or more of processing unit 110 and imageacquisition unit 120, with or without the need to access memory 140 or150. The order and arrangement of steps in process 4300 is provided forpurposes of illustration. As will be appreciated from this disclosure,modifications may be made to process 4300 by, for example, adding,combining, removing, and/or rearranging the steps of process 4300.

As illustrated in FIG. 43A, process 4300 may include a step 4302 ofacquiring an image representative of an environment of the vehicle. Inone exemplary embodiment, image acquisition unit 120 may acquire one ormore images of an area forward of vehicle 200 (or to the sides or rearof a vehicle, for example). For example, image acquisition unit 120 mayobtain an image using image capture device 122 having a field of view202. In other exemplary embodiments, image acquisition unit 120 mayacquire images from one or more of image capture devices 122, 124, 126,having fields of view 202, 204, 206. Image acquisition unit 120 maytransmit the one or more images to processing unit 110 over a dataconnection (e.g., digital, wired, USB, wireless, Bluetooth, etc.).

Process 4300 may also include a step 4302 of determining currentlocation 4214 of vehicle 200. Processing unit 110 may receive the one ormore images from image acquisition unit 120. Processing unit 110 mayexecute monocular image analysis module 402 to analyze the plurality ofimages at step 4302, as described in further detail in connection withFIGS. 5B-5D. By performing the analysis, processing unit 110 may detecta set of features within the set of images, for example, one or morelandmarks. Processing unit 110 may use the landmarks and performprocesses similar to those discussed above, for example, in FIGS. 34-36to determine current location 4214 of vehicle 200.

Process 4300 may include a step 4306 of determining whether vehicle 200is located on first road segment 4304. Processing unit 110 may determinewhether vehicle 200 is located on first road segment 4304 in many ways.For example, processing unit may compare its current location 4214determined in, for example, step 4304 with predetermined road modeltrajectory 4212 to determine whether current position 4214 is located onpredetermined road model trajectory 4212. Processing unit may determinethat vehicle 200 is located on first road segment 4304 when currentposition 4214 is located on predetermined road model trajectory 4212. Inanother exemplary embodiment, processing unit 110 may use landmarks anddirectional indicators for the landmarks to determine whether a currentposition 4214 of vehicle 200 is located on road segment 4204. Forexample, as discussed above with respect to FIGS. 34-36, if adirectional indicator of a recognized landmark intersects withpredetermined road model trajectory 4212 (discussed above, e.g., inrelation to FIGS. 34-36), processing unit 110 may determine that currentlocation 4214 of vehicle 200 lies in road segment 4204. When processingunit 110 determines that vehicle 200 is located in road segment 4204(Step 4306: Yes), processing unit 110 may proceed to step 4308. Whenprocessing unit 110 determines, however, that vehicle 200 is not locatedon road segment 4204 (Step 4306: No), processing unit 110 may proceed tostep 4314 via process segment C.

In step 4308, processing unit 110 may determine whether vehicle 200 islocated in overlap segment 4220. Processing unit 110 may use processessimilar to those discussed above with respect to step 4306 to determinewhether vehicle 200 is located in overlap segment 4220. For example,processing unit 110 may determine whether a direction indicatorcorresponding to a recognized landmark intersects predetermined roadmodel trajectory 4212 in the portion of predetermined road modeltrajectory 4212 located between A and B in overlap segment 4220. Inanother exemplary embodiment, processing unit 110 may compare currentlocation 4214 of vehicle 200 with the mathematical representation ofpredetermined road model trajectory 4212 to determine whether vehicle200 is located in overlap segment 4220. In yet another exemplaryembodiment, processing unit 110 may determine a distance travelled byvehicle 200 along predetermined road model trajectory 4212 in first roadsegment 4204. Processing unit may determine the distance travelled usingprocesses similar to those discussed above with respect to FIGS. 37-39regarding navigation using tail alignment. Processing unit 110 maydetermine whether current location 4214 of vehicle 200 lies in overlapsegment 4220 based on the distance travelled by vehicle 200. Whenprocessing unit 110 determines that vehicle 200 is located withinoverlap segment 4220 (Step 4308: Yes), processing unit 110 may proceedto step 4320 via process segment D. When processing unit 110 determines,however, that vehicle 200 is not located within overlap segment 4220(Step 4308: No), processing unit 110 may proceed to step 4310.

Process 4300 may include a step 4310 of determining first ANR.Processing unit 110 may determine first ANR based on its determinationthat vehicle 200 is located in first road segment 4204 but not inoverlap segment 4220. In one exemplary embodiment, processing unit 110may obtain a mathematical representation (e.g. three-dimensionalpolynomial) of predetermined road model trajectory 4212. Processing unit110 may determine direction 4230 of predetermined road model trajectory4212 as a vector oriented tangentially to predetermined road modeltrajectory 4212 at current location 4214 of vehicle 200. For example,processing unit 110 may determine direction 4230 as a vector pointingalong a gradient of the mathematical representation of predeterminedroad model trajectory 4212 at current location 4214. Although the abovedescription assumes that current location 4214 of vehicle 200 is locatedon predetermined road model trajectory 4212, processing unit 110 mayperform processes similar to those discussed above with respect to FIGS.34-39 when vehicle 200 is not located on predetermined road modeltrajectory 4212. For example, processing unit may determine a transformrequired to move vehicle 200 to predetermined road model trajectory 4212before determining direction 4230 as discussed above.

Processing unit 110 may also determine a heading direction 4240 ofvehicle 200. For example, as illustrated in FIG. 42, processing unit 110may determine heading direction 4240 of vehicle 200 as the direction inwhich image capture device 122 may be oriented relative to a localcoordinate system associated with vehicle 200. In another exemplaryembodiment, processing unit 200 may determine heading direction 4240 asthe direction of motion of vehicle 200 at current location 4214. In yetanother exemplary embodiment, processing unit may determine headingdirection 4240 based on a travelled trajectory as discussed above withrespect to FIGS. 37-39. Processing unit 110 may determine a rotationalangle □₁ between heading direction 4240 and direction 4230 ofpredetermined road model trajectory 4212. In one exemplary embodiment,first ANR may include rotation angle 1 that may help ensure that headingdirection 4240 of vehicle 200 may be aligned with direction 4230 ofpredetermined road model trajectory 4212. In another exemplaryembodiment, first ANR may also include accelerations or decelerations ofvehicle 200 that may be required to help ensure that heading direction4240 of vehicle 200 may be aligned with direction 4230 of predeterminedroad model trajectory 4212 in a predetermined amount of time.

Process 4300 may also include a step 4312 of adjusting steering system240 based on first ANR. Processing unit 110 may be configured to executeinstructions stored in navigational response module 408 to trigger firstANR by, for example, turning the steering wheel of vehicle 200 toachieve a rotation of angle 1. Processing unit 110 may also executeinstructions stored in navigational response module 408 to controlthrottling system 220 and/or braking system 230 to appropriately controla speed of vehicle 200 to help ensure that heading direction 4240 ofvehicle 200 may be aligned with direction 4230 of predetermined roadmodel trajectory 4212 in a predetermined amount of time.

Returning to step 4306, when processing unit 110 determines that vehicle200 is not located on road segment 4204 (Step 4306: No), processing unit110 may proceed to step 4314 via process segment C. In step 4314,processing unit 110 may determine whether vehicle 200 is located in roadsegment 4206. Processing unit 110 may perform operations similar tothose discussed above in step 4306 to determine whether vehicle 200 islocated in road segment 4206. When processing unit 110 determines thatvehicle 200 is not located in road segment 4206, process 4300 may end.When processing unit 110 determines, however, that vehicle 200 islocated in road segment 4206, processing unit 100 may proceed to step4316 of determining second ANR.

Processing unit 110 may determine second ANR using processes similar tothose discussed above with respect to step 4310. For example, processingunit 110 may determine a direction 4250 of predetermined road modeltrajectory 4226 at current location 4214 of vehicle 200, a headingdirection 4260, and an angle of rotation 2, which may help ensure thatheading direction 4260 of vehicle 200 may be aligned with direction4250. Further, like first ANR, second ANR may also include accelerationsor decelerations of vehicle 200 that may be required to help ensure thatheading direction 4260 of vehicle 200 may be aligned with direction 4250of predetermined road model trajectory 4226 in a predetermined amount oftime.

Process 4300 may also include a step 4318 of adjusting steering system240 based on second ANR. Processing unit 110 may be configured toexecute instructions stored in navigational response module 408 totrigger second ANR by, for example, turning the steering wheel ofvehicle 200 to achieve a rotation of angle 2. Processing unit 110 mayalso execute instructions stored in navigational response module 408 tocontrol throttling system 220 and/or braking system 230 to appropriatelycontrol a speed of vehicle 200 to help ensure that heading direction4260 of vehicle 200 may be aligned with direction 4250 of predeterminedroad model trajectory 4226 in a predetermined amount of time.

Returning to step 4308, when processing unit 110 determines that vehicle200 is located on overlap segment 4220 (Step 4308: Yes), processing unit110 may proceed to step 4320 via process segment D. In step 4320,processing unit 110 may determine first ANR. Processing unit 110 maydetermine first ANR using operations similar to those discussed abovewith respect to step 4310. Thus, for example, processing unit maydetermine a direction 4240 of predetermined road model trajectory 4212at current location 4214 of vehicle 200, a heading direction 4230, andan angle of rotation 1, which may help ensure that heading direction4240 of vehicle 200 may be aligned with direction 4230. Further, firstANR may also include accelerations or decelerations of vehicle 200 thatmay be required to help ensure that heading direction 4240 of vehicle200 may be aligned with direction 4230 of predetermined road modeltrajectory 4212 in a predetermined amount of time.

Process 4300 may also include a step 4322 of determining a second ANR.Processing unit 110 may determine second ANR using operations similar tothose discussed above with respect to step 4316. Thus, for example,processing unit may determine a direction 4260 of predetermined roadmodel trajectory 4226 at current location 4214 of vehicle 200, a headingdirection 4250, and an angle of rotation 2, which may help ensure thatheading direction 4260 of vehicle 200 may be aligned with direction4250. Further, second ANR may also include accelerations ordecelerations of vehicle 200 that may be required to help ensure thatheading direction 4260 of vehicle 200 may be aligned with direction 4250of predetermined road model trajectory 4226 in a predetermined amount oftime.

Process 4300 may also include a step 4324 of determining an errorbetween first ANR and second ANR. In one exemplary embodiment,processing unit 110 may determine the error as an error between anglesof rotation 1 and 2 determined, for example, in steps 4320 and 4322. Inanother exemplary embodiment, processing unit 110 may determine theerror as an error between direction 4230 of predetermined road modeltrajectory 4212 and direction 4250 of predetermined road modeltrajectory 4226. In another exemplary embodiment, processing unit 110may determine the error as a cosine distance between directions 4230 and4250. One of ordinary skill in the art would recognize that processingunit 110 may use other mathematical functions to determine the errorbetween directions 4230 and 4250.

Process 4300 may also include a step 4326 of determining whether theerror is less than a threshold error. Because processing unit 110 mayperform step 4324 only when vehicle 200 is located in overlap segment4220, the error may indicate whether the co-ordinate frame of vehicle200 is aligned with both road segments 4204 and 4206. It is contemplatedthat in some embodiments when vehicle 200 first enters overlap segment4220, the error may exceed the threshold error and navigating vehicle200 based on both navigational maps 4200 and 4202 may improve accuracy.As vehicle 200 travels further within overlap segment 4220, the errormay decrease and may eventually become less than the threshold error.When the co-ordinate frame of vehicle 200 is aligned with both roadsegments 4204 and 4206, the error may be smaller than the thresholderror and it may be sufficient to start navigating vehicle 200 basedonly on navigational map 4202.

When processing unit 110 determines that the error is greater than thethreshold error (Step 4326: Yes), processing unit 110 may proceed tostep 4328. In step 4328, processing unit 110 may determine third ANRbased on both the first ANR and the second ANR so that vehicle 200 maybe navigated based on both maps 4200 and 4202. Thus, for example,processing unit 110 may determine a third angle of rotation 3 as acombination of angles of rotation 1 and 2 determined, for example, insteps 4320 and 4322. In some exemplary embodiments, the combination maybe an average, a weighted average, or some other mathematicalcombination of angles of rotation 1 and 2. Likewise, processing unit 110may determine accelerations or decelerations for vehicle 200 based on acombination of the accelerations and/or decelerations determined, forexample, in steps 4320 and 4322.

Process 4300 may also include a step 4330 of adjusting steering system240 based on third ANR. Processing unit 110 may be configured to executeinstructions stored in navigational response module 408 to trigger thirdANR by, for example, turning the steering wheel of vehicle 200 toachieve a rotation of angle 3. Processing unit 110 may also executeinstructions stored in navigational response module 408 to controlthrottling system 220 and/or braking system 230 based on theaccelerations and/or decelerations determined in steps 4330 or 4332.

Returning to step 4326, when processing unit 110 determines that theerror is less than the threshold error (Step 4326: No), processing unit110 may proceed to step 4332 of determining the third ANR based only onthe second ANR. As discussed above, when the error is less than thethreshold error, it may be sufficient to navigate vehicle 200 based onlyon map 4202. Thus, in one exemplary embodiment, processing unit 110 mayset third ANR equal to second ANR. In another exemplary embodiment,processing unit 110 may set third ANR by scaling (i.e. magnifying orattenuating) second ANR using a scaling factor. After completing step4332, processing unit 110 may proceed to step 4330 of adjusting steeringsystem 240 based on third ANR.

Processing unit 110 and/or image acquisition unit 120 may repeat process4300 after a predetermined amount of time. In one exemplary embodiment,the predetermined amount of time may range between about 0.5 seconds to1.5 seconds. By repeatedly determining a current location 4214 ofvehicle 200, determining whether current location 4214 lies in overlapsegment 4220, and determining first ANR, second ANR, and third ANR basedon the location of vehicle 200, processing unit 110 and/or imageacquisition unit 120 may help to navigate vehicle 200, using overlappingroad segment 4220 of local maps 4200, 4202.

Sparse Map Autonomous Vehicle Navigation

In some embodiments, the disclosed systems and methods may use a sparsemap for autonomous vehicle navigation. As discussed above regardingFIGS. 8-11D, the sparse map may provide sufficient information fornavigation without requiring excessive data storage or data transferrates. Further, a vehicle (which may be an autonomous vehicle) may usethe sparse map to navigate one or more roads. For example, as discussedbelow in further detail, vehicle 200 may determine an autonomousnavigational response based on analysis of the sparse map and at leastone image representative of an environment of vehicle 200.

In some embodiments, vehicle 200 may access a sparse map that mayinclude data related to a road on which vehicle 200 is traveling andpotentially landmarks along the road that may be sufficient for vehiclenavigation. As described in sections above, the sparse data mapsaccessed by vehicle 200 may require significantly less storage space anddata transfer bandwidth as compared with digital maps including detailedmap information, such as image data collected along a road. For example,rather than storing detailed representations of a road segment on whichvehicle 200 is traveling, the sparse data map may store threedimensional polynomial representations of preferred vehicle paths alongthe road. A polynomial representation of a preferred vehicle path alongthe road may be a polynomial representation of a target trajectory alonga road segment. These paths may require very little data storage space.

Consistent with disclosed embodiments, an autonomous vehicle system mayuse a sparse map for navigation. As discussed earlier, at the core ofthe sparse maps, one or more three-dimensional contours may representpredetermined trajectories that autonomous vehicles may traverse as theymove along associated road segments. As also discussed earlier, thesparse maps may also include other features, such as one or morerecognized landmarks, road signature profiles, and any otherroad-related features useful in navigating a vehicle.

In some embodiments, an autonomous vehicle may include a vehicle bodyand a processor configured to receive data included in a sparse map andgenerate navigational instructions for navigating the vehicle along aroad segment based on the data in the sparse map.

As discussed above in connection with FIG. 8, vehicle 200 (which may bean autonomous vehicle) may access sparse map 800 to navigate. As shownin FIG. 8, in some embodiments, sparse map 800 may be stored in amemory, such as memory 140 or 150. For example, sparse map 800 may bestored on a storage device or a non-transitory computer-readable mediumprovided onboard vehicle 200 (e.g., a storage device included in anavigation system onboard vehicle 200). A processor (e.g., processingunit 110) provided on vehicle 200 may access sparse map 4400 stored inthe storage device or computer-readable medium provided onboard vehicle200 in order to generate navigational instructions for guiding theautonomous vehicle 200 as it traverses a road segment.

In some embodiments, sparse map 800 may be stored remotely. FIG. 44shows an example of vehicle 200 receiving data from a remote server4400, consistent with disclosed embodiments. As shown in FIG. 44, remoteserver 4400 may include a storage device 4405 (e.g., a computer-readablemedium) provided on remote server 4400 that communicates with vehicle200. For example, remote server 4400 may store a sparse map database4410 in storage device 4405. Sparse map database 4410 may include sparsemap 800. In some embodiments, sparse map database 4410 may include aplurality of sparse maps. Sparse map database 4410 may be indexed basedon certain regions (e.g., based on geographical boundaries, countryboundaries, state boundaries, etc.) or based on any appropriateparameter (e.g., type or size of vehicle, climate, etc.). Vehicle 200may communicate with remote server 440 via one or more networks (e.g.,over a cellular network and/or the Internet, etc.) through a wirelesscommunication path. In some embodiments, a processor (e.g., processingunit 110) provided on vehicle 200 may receive data included in sparsemap database 4410 over one or more networks from remove server 4400.Furthermore, vehicle 200 may execute instructions for navigating vehicle200 using sparse map 800, as discussed below in further detail.

As discussed above in reference to FIG. 8, sparse map 800 may includerepresentations of a plurality of target trajectories 810 for guidingautonomous driving or navigation along a road segment. Such targettrajectories may be stored as three-dimensional splines. The targettrajectories stored in sparse map 800 may be determined based on two ormore reconstructed trajectories of prior traversals of vehicles along aparticular road segment. A road segment may be associated with a singletarget trajectory or multiple target trajectories. For example, on a twolane road, a first target trajectory may be stored to represent anintended path of travel along the road in a first direction, and asecond target trajectory may be stored to represent an intended path oftravel along the road in another direction (e.g., opposite to the firstdirection). Additional target trajectories may be stored with respect toa particular road segment.

Sparse map 800 may also include data relating to a plurality ofpredetermined landmarks 820 associated with particular road segments,local maps, etc. As discussed in detail in other sections, theselandmarks may be used in navigation of vehicle 200. For example, in someembodiments, the landmarks may be used to determine a current positionof vehicle 200 relative to a stored target trajectory. With thisposition information, vehicle 200 may be able to adjust a headingdirection to match a direction of the target trajectory at thedetermined location.

Landmarks may include, for example, any identifiable, fixed object in anenvironment of at least one road segment or any observablecharacteristic associated with a particular section of a particular roadsegment. In some cases, landmarks may include traffic signs (e.g., speedlimit signs, hazard signs, etc.). In other cases, landmarks may includeroad characteristic profiles associated with a particular section of aroad segment. Further examples of various types of landmarks arediscussed in previous sections, and some landmark examples are shown anddiscussed above in connection with FIG. 10.

FIG. 45 shows vehicle 200 navigating along a multi-lane road consistentwith disclosed embodiments. Here, a vehicle 200 may navigate roadsegments present within a geographic region 1111 shown previously inFIG. 11B. As previously discussed in relation to FIG. 11B, road segment1120 may include a multilane road with lanes 1122 and 1124, doubleyellow line 1123, and branching road segment 1130 that intersects withroad segment 1120. Geographic region 1111 may also include other roadfeatures, such as a stop line 1132, a stop sign 1134, a speed limit sign1136, and a hazard sign 1138.

FIG. 46 shows vehicle 200 navigating using target trajectories along amulti-lane road consistent with disclosed embodiments. A vehicle 200 maynavigate geographic region 1111 shown previously in FIG. 11B and FIG.45, using target trajectory 4600. Target trajectory 4600 may be includedin a local map (e.g., local map 1140 of FIG. 11C) of sparse map 800, andmay provide a target trajectory for one or more lanes associated with aroad segment. As previously discussed, sparse map 800 may includerepresentations of road-related features associated with geographicregion 1111, such as representations of one or more landmarks identifiedin geographic region 1111. Such landmarks may include speed limit sign1136 and hazard sign 1138. Vehicle 200 may use speed limit sign 1136 andhazard sign 1138 to assist in determining its current location relativeto target trajectory 4600. Based on the determined current location ofvehicle 200 relative to target trajectory 4600, vehicle 200 may adjustits heading to match a direction of the target trajectory at thedetermined location.

As discussed above, in some embodiments, sparse may 800 may also includeroad signature profiles. Such road signature profiles may be associatedwith any discernible/measurable variation in at least one parameterassociated with a road. For example, in some cases, such profiles may beassociated with variations in surface roughness of a particular roadsegment, variations in road width over a particular road segment,variations in distances between dashed lines painted along a particularroad segment, variations in road curvature along a particular roadsegment, etc.

FIG. 47 shows an example of a road signature profile 1160 associatedwith vehicle 200 as it travels on the road shown in FIGS. 45 and 46.While profile 1160 may represent any of the parameters mentioned above,or others, in relation to vehicle 200, in one example, profile 1160 mayrepresent a measure of road surface roughness obtained by monitoring oneor more sensors providing outputs indicative of an amount of suspensiondisplacement as a vehicle 200 travels a road segment in FIG. 46.Alternatively, profile 1160 may represent variation in road width, asdetermined based on image data obtained via a camera onboard vehicle 200traveling in a road segment in FIG. 46. Such profiles may be useful, forexample, in determining a particular location of vehicle 200 relative totarget trajectory 4600, and may aid in navigation of vehicle 200. Thatis, as vehicle 200 traverses a road segment of FIG. 46, vehicle 200 maymeasure a profile associated with one or more parameters associated withthat road segment. If the measured profile can be correlated/matchedwith a predetermined profile that plots the parameter variation withrespect to position along the road segment, then the measured andpredetermined profiles may be used by vehicle 200 (e.g., by overlayingcorresponding sections of the measured and predetermined profiles) inorder to determine a current position along the road segment and,therefore, a current position relative to target trajectory 4600 for theroad segment. Measurements of the profile by vehicle 200 may continue asvehicle 200 travels in lane 1124 of FIG. 46 in order to continuouslydetermine a current position along the road segment and a currentposition of vehicle 200 relative to target trajectory 4600. As such,navigation of vehicle 200 may be provided.

FIG. 48 is an illustration of an example of a portion of a roadenvironment 4800, as shown in FIGS. 45 and 46. In this example, FIG. 48shows road segment 1120. Vehicle 200 may be traveling along road segment1120. Along the road segment 1120, landmarks such as speed limit sign1136 and hazard sign 1138 may be present. Speed limit sign 1136 andhazard sign 1138 may be recognized landmarks that are stored in sparsemap 800, and may be used for autonomous vehicle navigation along roadsegment 1120 (e.g., for locating vehicle 200, and/or for determining atarget trajectory of vehicle 200). Recognized landmarks 1136 and 1138 insparse map 800 may be spaced apart from each other at a certain rate.For example, recognized landmarks may be spaced apart in the sparse mapat a rate of no more than 0.5 per kilometer, at a rate of no more than 1per kilometer, or at a rate of no more than 1 per 100 meters. Landmarks1136 and 1138 may be used, for example, to assist vehicle 200 indetermining its current location relative to target trajectory 4600,such that the vehicle may adjust its heading to match a direction of thetarget trajectory at the determined location.

FIG. 49 is a flow chart showing an exemplary process 4900 for sparse mapautonomous navigation consistent with the disclosed embodiments.Processing unit 110 may utilize one of or both of application processor180 and image processor 190 to implement process 4900. As discussedbelow in further detail, vehicle 200 may determine an autonomousnavigational response based on analysis of a sparse map and at least oneimage representative of an environment of vehicle 200.

At step 4902, processing unit 110 may receive a sparse map of a roadsegment, such as sparse map 800, from memory 140 or 150. For example,the sparse map may be transmitted to processing unit 110 based on acalculation of the position of vehicle 200 by position sensor 130. Inother exemplary embodiments, vehicle 200 may receive the sparse map fromremote server 4400. The sparse map data may have a particular datadensity. The data density of the sparse map may be expressed in terms ofdata unit per unit distance. For example, the sparse map may have a datadensity of no more than 1 megabyte per kilometer. In another example,the sparse map may have a data density of no more than 100 kilobytes perkilometer. In another example, the sparse map may have a data density ofno more than 10 kilobytes per kilometer. Data density may be expressedin terms of any conceivable data unit and unit distance. Further, thesparse map may include a polynomial representation of a targettrajectory along the road segment.

At step 4904, processing unit 110 may receive at least one imagerepresentative of an environment of vehicle 200. For example, processingunit 110 may receive at least one image from image acquisition unit 120using image capture device 122. In other exemplary embodiments, imageacquisition unit 120 may acquire one or more images from one or more ofimage capture devices 122, 124, and 126. Image acquisition unit 120 maytransmit the one or more images to processing unit 110 over a dataconnection (e.g., digital, wired, USB, wireless, Bluetooth, etc.).

At step 4906, processing unit 110 may analyze the received sparse mapand the at least one image of the environment of vehicle 200. Forexample, processing unit 110 may execute monocular image analysis module402 to analyze one or more images, as described in further detail inconnection with FIGS. 5B-5D. By performing the analysis, processing unit110 may detect a set of features within the set of images, for example,one or more landmarks, such as landmarks 1134, 1136, and 1138. Asdiscussed earlier, landmarks may include one or more traffic signs,arrow markings, lane markings, dashed lane markings, traffic lights,stop lines, directional signs, reflectors, landmark beacons, lampposts,a change is spacing of lines on the road, signs for businesses, and thelike. Furthermore, processing unit 110 may analyze the sparse map todetermine that an object in one or more images is a recognized landmark.For example, processing unit 110 may compare the image of the object todata stored in the sparse map. Based on the comparison, the imageprocessor 190 may determine whether or not the object is a recognizedlandmark. Processing unit 110 may use recognized landmarks from capturedimage data of the environment and/or UPS data to determine a position ofvehicle 200. Processing unit 110 may then determine a position ofvehicle 200 relative to a target trajectory of the sparse map.

At step 4908, processing unit 110 may cause one or more navigationalresponses in vehicle 200 based solely on the analysis of the sparse mapand at least one image of the environment performed at step 4906. Forexample, processing unit 110 may select an appropriate navigationalresponse based on the position of vehicle 200 relative to the targettrajectory of the sparse map. Navigational responses may include, forexample, a turn, a lane shift, a change in acceleration, and the like.Processing unit 110 may cause system 100 to provide inputs (e.g.,control signals) to one or more of throttling system 220, braking system230, and steering system 240 as shown in FIG. 2F to navigate vehicle 200(e.g., by causing an acceleration, a turn, a lane shift, etc.) toprovide a navigational response. System 100 may provide inputs to one ormore of throttling system 220, braking system 230, and steering system240 over one or more data links (e.g., any wired and/or wireless link orlinks for transmitting data). Additionally, multiple navigationalresponses may occur simultaneously, in sequence, or any combinationthereof. For instance, processing unit 110 may cause vehicle 200 toshift one lane over and then accelerate by, for example, sequentiallytransmitting control signals to steering system 240 and throttlingsystem 220 of vehicle 200. Alternatively, processing unit 110 may causevehicle 200 to brake while at the same time shifting lanes by, forexample, simultaneously transmitting control signals to braking system230 and steering system 240 of vehicle 200.

Navigation Based on Expected Landmark Location

Landmarks appearing in one or more images captured by a camera onboard avehicle may be used in the disclosed embodiments to determine a locationof a vehicle along a road model trajectory. Such landmarks may includerecognized landmarks represented, for example, in sparse map 800.Processing unit 110 of vehicle 200 may analyze images captured from oneor more cameras onboard vehicle 200 to look for and verify the presenceof a recognized landmark (from sparse data map 800) in the capturedimages. According to techniques described in detail in other sections ofthe disclosure, the verified, recognized landmarks in the environment ofthe vehicle can then be used to navigate the vehicle (e.g., by enablinga determination of a position of vehicle 200 along a target trajectoryassociated with a road segment).

In the disclosed embodiments, however, processor unit 110 may alsogenerate navigational instructions on not only those landmarks appearingin captured images, but also based on an expected location of therecognized landmark as conveyed by sparse data map 800. For example,braking of a vehicle may be initiated a certain distance from recognizedlandmarks such as a stop line, a traffic light, a stop sign, a sharpcurve, etc., even before those landmarks are detectable via an on-boardcamera. Landmarks may include, for example, any identifiable, fixedobject in an environment of at least one road segment or any observablecharacteristic associated with a particular section of the road segment.In some cases, landmarks may include traffic signs (e.g., speed limitsigns, hazard signs, etc.). In other cases, landmarks may include roadcharacteristic profiles associated with a particular section of a roadsegment. Further examples of various types of landmarks are discussed inprevious sections, and some landmark examples are shown in FIG. 10.

FIG. 50 illustrates an example environment consistent with the disclosedembodiments. Vehicle 200 (which may be an autonomous vehicle) may travelalong a target road model trajectory 5002 in road 5004. Vehicle 200 maybe equipped with one or more image capture devices (e.g., one or more ofimage capture device 122, 124, or 126) that capture an image of theenvironment of the vehicle. The one or more image capture devices mayhave a sight range 5006. Sight range 5006 may define a range at which animage capture device of vehicle 200 can capture accurate images of theenvironment around vehicle 200. For example, sight range 5006 may definethe range at which the field of view, focal length, resolution focus,sharpness, image quality, and the like of the image capture device ofvehicle 200 is sufficient to provide images for navigation of vehicle200. Region 5008 may define a range outside of the sight range 5006 ofan image capture device of vehicle 200. In region 5008, an image capturedevice of vehicle 200 may not be able to capture images of theenvironment around vehicle 200 that are sufficient to allow navigationof vehicle 200. In other exemplary embodiments, each image capturedevice may have a different sight range.

As shown in FIG. 50, recognized landmark 5010 is within sight range5006. Because recognized landmark 5010 is within sight range 5006, itmay be captured by an image capture device of vehicle 200 andidentified, and used to navigate vehicle 200. Recognized landmark 5010may be identified by vehicle 200 according to techniques discussed abovein connection with, for example, FIGS. 34-36.

As previously discussed, recognized landmark 5012 is within region 5008.However, region 5008 defines a range outside of the sight range 5006 ofan image capture device of vehicle 200. Accordingly, vehicle 200 may notbe able to identify recognized landmark 5012 using an image capturedevice of vehicle 200 because recognized landmark 5012 is out of thesight range of the image capture device.

Consistent with disclosed embodiments, vehicle 200 may identifyrecognized landmark 5012 using alternative techniques. For example, animage capture device of vehicle 200 may capture an image of theenvironment within sight range 5006. A processor of vehicle 200 (e.g.,processing unit 110) may receive the image. The processor may thendetermine a position of vehicle 200 along predetermined road modeltrajectory 5002 in road 5004 based on the captured image. For example,as discussed in other sections, the processor may compare datainformation representing recognized landmark 5010 from the capturedimage of the environment to stored data, such as data stored in sparsemap 800, discussed above, to determine a position of vehicle 200 alongpredetermined road model trajectory 5002 in road 5004.

Based on the determined position of vehicle 200, the processor may thenidentify a recognized landmark beyond sight range 5006 (e.g., recognizedlandmark 5012) forward of the vehicle 200. For example, by accessinginformation stored in sparse data map 800 or any portion of sparse datamap 800 (e.g., any received local map portions of sparse data map 800)processing unit 110 of vehicle 200 may determine the next expectedrecognized landmark to be encountered by vehicle 200 (or any otherrecognized landmark to be encountered by vehicle 200). The processor mayalso determine a predetermined position of recognized landmark 5012based on the information available in sparse data map 800. Then,processing unit 110 may determine a current distance 5014 between thevehicle 200 and expected, recognized landmark 5012. The current distance5014 between the vehicle 200 and the recognized landmark 5012 may bedetermined by comparing the determined position of vehicle 200 with thepredetermined position of recognized landmark 5012. Based on thedistance 5014, the processor of vehicle 200 may then determine anautonomous navigational response for the vehicle. For example, amongother responses, processing unit 110 may initiate braking in advance oflandmark 5012 even prior to detection of landmark 5012 in any capturedimages from image capture devices onboard vehicle 200.

FIG. 51 illustrates a configuration 5100 for autonomous navigationconsistent with disclosed embodiments. As discussed earlier, processingunit 110 may receive images from an image acquisition unit 120. Imageacquisition unit may include one or more image capture devices (e.g.,image capture device 122, 124, or 126). The images may depict anenvironment of vehicle 200 within the field of view of an image capturedevice onboard vehicle 200.

While GPS data need not be relied upon to determine an accurate positionof vehicle 200 along a target trajectory, GPS data (e.g., GPS data fromGPS unit 5106) may be used as an index for determining relevant localmaps to access from within sparse data map 800. Such GPS data may alsobe used as a general index to aid in verifying an observed recognizedlandmark.

FIG. 52 shows an example of an environment 5200 consistent with thepresent disclosure. As shown in FIG. 52, a vehicle 200 may approach ajunction 5202 with a stop sign 5204 and a stop line 5210. One or both ofstop sign 5204 or stop line 5210 may correspond to recognized landmarksrepresented in sparse data map 800. Either or both of stop sign 5204 orstop line 5210 may be located in a region 5208 beyond a focal length ofan image capture device aboard vehicle 200 or otherwise outside of ausable sight range of the image capture device. Based on informationstored in sparse data map 800 relative to stop sign 56204 and/or stopline 5210, processing unit 110 may initiate braking based on adetermined, expected distance to stop line 5204 or stop line 5210 evenbefore stop line 5204 or 5210 have been identified in images receivedfrom the image capture device onboard vehicle 200. Such a navigationtechnique, for example, may aid in slowing vehicle 200 gradually oraccording to a predetermined braking profile even without visualconfirmation of a distance to a trigger for braking (e.g., stop sign5204, stop line 5210, an expected curve, etc.).

FIG. 53 shows another example environment 5300 consistent with thepresent disclosure. As shown in FIG. 53, a vehicle 200 may approach acurve 5302 of road 5304. Vehicle 200 may include an image acquisitionunit (e.g., image acquisition unit 120) including one or more imagecapture devices that provide a sight range of 5306. Region 5308 maydefine a range outside of the sight range 5306 of the image acquisitionunit of vehicle 200.

Vehicle 200 may need to slow down in speed or implement steering toaccount for curve 5302 in road 5304. To plan a slowdown in speed orimplement steering, it may be useful to know in advance where the curve5302 is located. However, curve 5302 may be located in region 5308,which is beyond the focal length of an image capture device aboardvehicle 200. Thus, vehicle 200 may use a predetermined position of curve5302, for example, as represented in sparse data map 800, as well as theposition of vehicle 200 along predetermined road model trajectory 5310,to determine a distance 5320 to curve 5302. This distance may be used toslow vehicle 200 change a course of vehicle 200, etc. before the curveappears in images captured by an onboard camera.

Consistent with disclosed embodiments, to determine distance 5320 tocurve 5302, the image acquisition device of vehicle 200 may capture animage of the environment. The image may include a recognized landmark5318. A processor of vehicle 200 (e.g., processing unit 110) may receivethe image and determine a position of vehicle 200 along predeterminedroad model trajectory 5310 based on the captured image and the positionof recognized landmark 5318. Based on the determined position of vehicle200, the processor may then identify curve 5302 beyond sight range 5306forward of the vehicle 200 based on information included in sparse datamap 800 relevant to curve 5302. Position information included in sparsedata map 800 for curve 5302 may be compared with a determined positionfor vehicle 200 along a target trajectory for vehicle 200 to determine adistance between vehicle 200 and curve 5302. This distance can be usedin generating a navigational response for vehicle 200 prior toidentification of curve 5302 within images captured by a camera onboardvehicle 200.

FIG. 54 is a flow chart showing an exemplary process 5400 forautonomously navigating vehicle 200 consistent with the disclosedembodiments. A processing unit (e.g., processing unit 110) of vehicle200 may use one of or both of application processor 180 and imageprocessor 190 to implement process 5400. As discussed below in furtherdetail, vehicle 200 may autonomously navigate along a road segment basedon a predetermined landmark location. Furthermore, the predeterminedlandmark location may be beyond a sight range of vehicle 200.

At step 5402, a processing unit (e.g., processing unit 110) of vehicle200 may receive at least one image from an image capture device (e.g.,image capture device 122) of vehicle 200. The at least one image may berepresentative of an environment of vehicle 200. The at least one imagemay include data representative of one or more landmarks in theenvironment. For example, the at least one image may include datarepresentative of landmarks such as road signs (including stop signs,yield signs, and the like), traffic lights, general signs, lines on theroad, and curves along a road segment. As discussed in previoussections, a processing unit of vehicle 200 may verify recognizedlandmarks that appear in the at least one image.

At step 5404, the processing unit of vehicle 200 may determine aposition of vehicle 200. For example, the processing unit of vehicle 200may determine a position of vehicle 200 along a predetermined road modeltrajectory associated with a road segment based, at least in part, oninformation associated with the at least one image.

At step 5406, a recognized landmark beyond the focal range of the imagecapture device of vehicle 200 and forward of vehicle 200 may beidentified. The identification may be based on the determined positionof vehicle 200 along the predetermined road model trajectory associatedwith the road segment. For example, information about recognizedlandmarks along a predetermined road model trajectory may be previouslystored in a sparse map, such as sparse map 800, discussed above. Basedon the determined position of vehicle 200 along the predetermined roadmodel trajectory associated with the road segment, the processing unitof vehicle 200 may determine that one or more recognized landmarks arelocated forward of vehicle 200 along the predetermined road modeltrajectory, but beyond a sight range of the image capture device ofvehicle 200. Moreover, the processing unit of vehicle 200 may access apredetermined position of the recognized landmarks by accessing sparsemap 800.

At step 5408, a current distance between the vehicle and the recognizedlandmark located forward of vehicle 200 beyond a sight range of theimage capture device of vehicle 200 may be determined. The currentdistance may be determined by comparing the determined position ofvehicle 200 along the predetermined road model trajectory associatedwith the road segment to the predetermined position of the recognizedlandmark forward of vehicle 200 beyond the sight range.

At step 5410, an autonomous navigational response for the vehicle may bedetermined based on the determined current distance between vehicle 200and the recognized landmark located forward of vehicle 200. Processingunit 110 may control one or more of throttling system 220, brakingsystem 230, and steering system 240 to perform a certain navigationalresponse, as discussed in other sections of this disclosure. Forexample, an autonomous navigational response may include sending acontrol signal to braking system 230 to provide the application ofbrakes associated with vehicle 200. In another example, an autonomousnavigational response may include sending a control signal to steeringsystem 240 to modify a steering angle of vehicle 200.

Autonomous Navigation Based on Road Signatures

Consistent with disclosed embodiments, the system may navigate based onpredetermined road signatures without using landmarks. As discussedabove, such road signatures may be associated with any discernible ormeasurable variation in at least one parameter associated with a road.For example, in some cases, road signatures may be associated withvariations in surface roughness of a particular road segment, variationsin road width over a particular road segment, variations in distancesbetween dashed lines painted along a particular road segment, variationsin road curvature along a particular road segment, etc. The roadsignatures may be identified as a vehicle traverses a road segment basedon visual information (e.g., images obtained from a camera) or based onother sensor output (e.g., one or more suspension sensor outputs,accelerometers, etc.). These signatures may be used to locate thevehicle along a predetermined road profile, and the forward trajectorycan then be determined for the vehicle based on the direction of theroad model at the determined location compared to a heading directionfor the vehicle.

FIG. 55 is a diagrammatic representation of exemplary vehicle controlsystems, consistent with the disclosed embodiments. As illustrated inFIG. 55, vehicle 200 (which may be an autonomous vehicle) may includeprocessing unit 110, which may have features similar to those discussedabove with respect to FIGS. 1 and 2F. Vehicle 200 may also includeimaging unit 220, which may also have features similar to thosediscussed above with respect to FIGS. 1 and 2F. In addition, vehicle 200may include one or more suspension sensors 5500 capable of detectingmovement of the suspension of vehicle 200 relative to a road surface.For example, signals from suspension sensors 5500 located adjacent eachwheel of vehicle 200 may be used to determine a local shape,inclination, or banking of the road surface over which vehicle 200 maybe located. In some exemplary embodiments, vehicle 200 may additionallyor alternatively include accelerometers or other position sensors thatmay acquire information regarding variations in the road surface asvehicle 200 travels over the road surface. It is also contemplated thatsystem 100 illustrated in FIG. 55 may include some or all of thecomponents described above with respect to, for example, FIGS. 1 and 2F.

FIG. 56 illustrates vehicle 200 travelling on road segment 5600 in whichthe disclosed systems and methods for navigating vehicle 200 using oneor more road signatures may be used. Road segment 5600 may include lanes5602 and 5604. As illustrated in FIG. 56, lane 5602 may be delimited byroad center 5606 and right side 5608, whereas lane 5604 may be delimitedby left side 5610 and road center 5606. Lanes 5602 and 5604 may have thesame or different widths. It is also contemplated that each of lanes5602, 5604 may have uniform or non-uniform widths along a length of roadsegment 5600. Although FIG. 56 depicts road segment 5600 as includingonly two lanes 5602, 5604, it is contemplated that road segment 5600 mayinclude any number of lanes.

In one exemplary embodiment as illustrated in FIG. 56, vehicle 200 maytravel along lane 5602. Vehicle 200 may be configured to travel alongpredetermined road model trajectory 5612, which may define a preferredpath (e.g., a target road model trajectory) within lane 5602 of roadsegment 5600. In some exemplary embodiments, predetermined road modeltrajectory 5612 may be located equidistant from road center 5606 andright side 5608. It is contemplated however that predetermined roadmodel trajectory 5612 may be located nearer to one or the other ofcenter 5606 and right side 5608 of road segment 5600. In someembodiments, road model trajectory 5612 may be located elsewhere withrespect to the road. For example, road model trajectory 5612 may belocated to approximately coincide with a center of a roadway, a roadedge, a lane edge, etc.

In some embodiments, predetermined road model trajectory 5612 may bemathematically represented by a three-dimensional polynomial function,which may be stored in memories 140, 150 associated with vehicle 200. Itis also contemplated that the three-dimensional polynomialrepresentation of road model trajectory 5612 may be stored in a storagedevice located remotely from vehicle 200. Processing unit 110 of vehicle200 may be configured to retrieve predetermined road model trajectory5612 from storage device over a wireless communications interface.

Vehicle 200 may be equipped with image capture devices 122, 124, 126 ofimage acquisition unit 120. It is contemplated that vehicle 200 mayinclude more or fewer image capture devices than those shown in FIG. 56.Image capture devices 122, 124, 126 may be configured to acquire aplurality of images representative of an environment of vehicle 200, asvehicle 200 travels along road segment 5600. For example, one or more ofimage capture devices 122, 124, 126 may obtain the plurality of imagesshowing views forward of vehicle 200. Processing unit 110 of vehicle 200may be configured to detect a location of vehicle 200 at vehicle travelsalong road segment 5600 based on the one or more images obtained byimage capture devices 122, 124, 126 or based on signals received from,for example, suspension sensor 5500.

As illustrated in FIG. 56, vehicle 200 may travel via locations 5622,5624, 5626, to current location 5628. Although only three priorlocations 5622-5626 are illustrated in FIG. 56, one of ordinary skill inthe art would recognize that any number of previous locations of vehicle200 may be present on road segment 5600. Processing unit 110 may analyzethe one or more images received from image capture devices 122, 124,126, to determine, for example, road widths W_(p1), W_(p2), W_(p3),W_(c) at locations 5622, 5624, 5626, 5628, respectively, where thesubscript “p” refers to a previous location and the subscript “c” refersto current location 5628 of vehicle 200. In some exemplary embodiments,processing unit 110 may additionally or alternatively determine, forexample, lane widths D_(p1), D_(p2), D_(p3), D_(c) at locations 5622,5624, 5626, 5628, respectively. Processing unit 110 may generate a roadwidth profile or a lane width profile over portion 5614 of road segment5600. The determined road width profile or lane width profile maycorrespond to current location 5628.

FIG. 57 illustrates an exemplary profile 5700 generated by processingunit 110 of vehicle 200. As illustrated in FIG. 57, road width, lanewidth, or other parameters may be charted on the y-axis against adistance travelled by vehicle 200 along road segment 5600 on the x-axis.Processing unit 110 may determine the distance travelled using systemsand methods similar to those discussed above with respect to FIGS.34-36.

Processing unit 110 may determine a local feature of road segment 5600corresponding to current location 5628 of vehicle 200. For example,processing unit 110 may determine a mathematical representation for theprofile (e.g. profile shown in FIG. 57) by curve fitting the determinedroad widths W_(p1), W_(p2), W_(p3), W_(c) and/or lane widths D_(p1),D_(p2), D_(p3), D_(c). In one exemplary embodiment, processing unit 110may determine, for example, coefficients (e.g. a₁, a₂, . . . a_(n))associated with the curve fit of the road width profile or the lanewidth profile. The determined coefficients may represent the localfeature of road segment 5600 at current location 5628. In anotherexemplary embodiment, processing unit 110 may determine a slope ofprofile 5700 as the local feature. It is contemplated that processingunit 110 may perform other mathematical operations on profile 5700 todetermine the local feature of road segment 5600 corresponding to acurrent location of vehicle 200.

Processing unit 110 may retrieve a predetermined signature featureassociated with road segment 5600, for example, from database 160 storedin memories 140, 150. In one exemplary embodiment, the predeterminedsignature features may include coefficients of best or preferred fitlines representing road width profiles or lane width profilescorresponding to various locations along predetermined road modeltrajectory 5612. For example, the predetermined signature features mayinclude coefficients b₁, b₂, . . . b_(n) at location 1; c₁, c₂, . . .c_(n) at location 2; d₁, d₂, . . . d_(n) at location 3, etc. Processingunit 110 may compare the coefficients (e.g. a₁, a₂, . . . a_(n))determined based on road widths W_(p1), W_(p2), W_(p3), W_(c) and/orlane widths D_(p1), D_(p2), D_(p3), D_(c) with the coefficients (e.g.b₁, b₂, . . . b_(n); c₁, c₂, . . . c_(n); d₁, d₂, . . . d_(n); etc.).Processing unit 110 may determine current location 5628 of vehicle 200based on a match of the coefficients. For example, if coefficients a₁,a₂, . . . a_(n) match with coefficients c₁, c₂, . . . c_(n),respectively, processing unit 110 may determine location 5628 of vehicle200 as corresponding to location 2 of predetermined road modeltrajectory 5612.

Processing unit 110 may determine a match in many ways. In one exemplaryembodiment, processing unit 110 may determine a distance measure betweenthe coefficients (e.g. a₁, a₂, . . . a_(n)) and each set of coefficients(e.g. b₁, b₂, . . . b_(n); c₁, c₂, . . . c_(n); d₁, d₂, . . . d_(n);etc.) corresponding to locations 1, 2, 3, etc. Processing unit 110 maydetermine that there is a match when at least one of the determineddistance measures is less than a threshold distance. In other exemplaryembodiments, processing unit 110 may determine an error between thecoefficients (e.g. a₁, a₂, . . . a_(n)) and each set of coefficients(e.g. b₁, b₂, . . . b_(n); c₁, c₂, . . . c_(n); d₁, d₂, . . . d_(n);etc.) corresponding to locations 1, 2, 3, etc. Processing unit 110 maydetermine a match when at least one error is less than a thresholderror. One of ordinary skill in the art would recognize that processingunit 110 may use other mathematical computations to determine acorrelation or match between the two sets of coefficients.

In some exemplary embodiments, processing unit 110 may use road width W₄and/or lane width w₄ as local feature to determine current location 5628of vehicle 500. For example, the predetermined signature features ofroad segment 5600 may include road widths w₁, w₂, w₃, w₄, w₅, . . .w_(n) corresponding to locations 1, 2, 3, 4, 5, . . . n alongpredetermined road model trajectory 5612. Additionally, oralternatively, the predetermined signature features of road segment 5600may include lane widths d₁, d₂, d₃, d₄, d₅, . . . d_(n) corresponding tolocations 1, 2, 3, 4, 5, . . . n along predetermined road modeltrajectory 5612. Processing unit 110 may compare road width W_(c) and/orlane width D_(c) with road widths w₁, w₂, w₃, w₄, w₅, . . . w_(n) and/orlane widths d₁, d₂, d₃, d₄, d₅, . . . d_(n), respectively, to determinecurrent location 5628. For example, if road width W_(c) matches withroad width w₅, processing unit 110 may determine location 5628 ascorresponding to location 5. Likewise, if lane width D_(c) matches withlane width d₃, processing unit 110 may determine location 5628 ascorresponding to location 3. Processing unit may determine whether roadwidth W₄ and/or lane width D₄ and match using matching techniquessimilar to those discussed above.

Processing unit 110 may use other parameters to determine currentlocation 5628. For example, processing unit 110 may determine one ormore of average road width W_(avg) (e.g. average of W_(p1), W_(p2),W_(p3), W_(c)), road width variance W_(var) (e.g. variance of W_(p1),W_(p2), W_(p3), W_(c)), average lane width D_(avg) (e.g. average ofD_(p1), D_(p2), D_(p3), D_(c)), lane width variance D_(var) (e.g.variance of D_(p1), D_(p2), D_(p3), D_(c)), or other parameters such asmedian, mode, etc. to represent the local feature corresponding tocurrent location 5628. The corresponding predetermined road signaturefeature may also be represented by average road widths, road widthvariances, average lane widths, lane width variances, median or modevalues of road widths, median or mode values of lane widths, etc., atpredetermined locations on predetermined road model trajectory 5612.Processing unit 110 may determine current location 5628 of vehicle 200by comparing the determined local feature and the predetermined roadsignature features as discussed above.

In some exemplary embodiments, the local features and predeterminedsignature features of road segment 5600 may be based on lengths of, orspacing between, marks on road (road markings) segment 5600. FIG. 58illustrates vehicle 200 travelling on road segment 5600 in which thepredetermined road signatures may be based on road markings on roadsegment 5600. For example, FIG. 58 illustrates road center 5606 as adashed line represented by road markings 5802-5816. As vehicle 200travels along road segment 5600, processing unit 110 may analyze the oneor more images received from the one or more image capture devices 122,124, 126, etc. to detect road markings 5802-5816. Processing unit 110may also determine, for example, spacings S_(p1), S_(p2), S_(p3), S_(c)between road markings 5802-5804, 5804-5806, 5806-5808, 5808-5810,respectively. Processing unit 110 may additionally or alternativelydetermine lengths L_(p1), L_(p2), L_(p3), L_(c) for road markings 5802,5804, 5806, 5808, respectively. In one exemplary embodiment, processingunit may generate a dashed line spacing profile or a dashed line lengthprofile based on spacings S_(p1), S_(p2), S_(p3), S_(c) or lengthsL_(p1), L_(p2), L_(p3), L_(c), respectively, in a manner similar to theprofiles discussed above with respect to FIGS. 56 and 57. Processingunit 110 may also determine a local feature based on coefficients ofcurve fits to the dashed line spacing profile and/or dashed line lengthprofile as discussed above with respect to FIGS. 56 and 57. Processingunit 110 may compare the local feature (e.g., coefficients representingthe dashed line spacing profile or dashed line length profile) topredetermined signature features of road segment 5600. For example,processing unit 110 may compare the coefficients representing thedetermined dashed line spacing profile or dashed line length profilewith predetermined coefficients of dashed line spacing/length profilesat known locations along predetermined road model trajectory. Processingunit 110 may determine current location 5628 of vehicle 200 when thecoefficients of the determined dashed line spacing/length profiles matchthe predetermined coefficients at a particular known location asdiscussed above with respect to FIGS. 56 and 57.

In some exemplary embodiments, processing unit 110 may use dashed linespacing S_(c) and/or dashed line length L_(c) as a local feature todetermine current location 5628 of vehicle 500. For example, thepredetermined signature features of road segment 5600 may include dashedlane spacings s₁, s₂, s₃, s₄, s₅, . . . s_(n) corresponding to locations1, 2, 3, 4, 5, . . . n along predetermined road model trajectory 5612.Additionally, or alternatively, the predetermined signature features ofroad segment 5600 may include dashed line lengths l₁, l₂, l₃, l₄, l₅, .. . l_(n) corresponding to locations 1, 2, 3, 4, 5, . . . n alongpredetermined road model trajectory 5612. Processing unit 110 maycompare dashed line spacing S_(c) and/or dashed line length L_(c) withdashed lane spacings s₁, s₂, s₃, s₄, s₅, . . . s_(n) and/or dashed linelengths l₁, l₂, l₃, l₄, l₅, . . . l_(n), respectively, to determinecurrent location 5628. For example, if dashed line spacing S_(c) matcheswith dashed line spacing s₅, processing unit 110 may determine location5628 as corresponding to location 5. Likewise, if dashed line lengthL_(c) matches with dashed line length l₃, processing unit 110 maydetermine location 5628 as corresponding to location 3. Processing unitmay determine whether dashed line spacing S_(c) and/or dashed lanelength L_(c) match the predetermined dashed line lengths or spacingsusing matching techniques similar to those discussed above.

In other exemplary embodiments, processing unit 110 may determine anaverage dash line length L_(avg), dash line variance L_(var), dash linespacing average S_(avg), or dash line spacing variance S_(var) as alocal parameter. Processing unit 110 may compare dash mark lengthL_(avg), dash mark variance 1L_(var), dash mark spacing average S_(avg),or dash mark spacing variance S_(var) with predetermined values of dashmark length, dash mark variance, dash mark spacing average, or dash markspacing variance at various locations along predetermined road modeltrajectory 5612. The predetermined values of dash mark length, dash markvariance, dash mark spacing average, or dash mark spacing variance atvarious locations may constitute predetermined signature features ofroad segment 5600. Processing unit 110 may determine current location5628 of vehicle 200 as the location for which at least one of dash marklength L_(avg), dash mark variance L_(var), dash mark spacing averageS_(avg), or dash mark spacing variance S_(var) matches a predeterminedcorresponding value of dash mark length, dash mark variance, dash markspacing average, or dash mark spacing.

In yet other exemplary embodiments, processing unit 110 may use a numberof dashed lines as a local feature. For example, road markings 5802-5816may be painted at a fixed length and spacing when they are painted by amachine. Thus, it may be possible to determine current location 5628 ofvehicle 200 based on a count of the road markings as vehicle 200 travelson road segment 5600. Processing unit 110 may determine a count “N_(c)”of dash marks that vehicle 200 may have passed till it reaches currentlocation 5628. Processing unit 110 may compare count N_(c) with countsn₁, n₂, n₃, . . . n_(n) corresponding to the number of road markings upto locations 1, 2, 3, . . . n, respectively, along predetermined roadmodel trajectory 5612. Counts n₁, n₂, n₃, . . . n_(n) may correspond tothe predetermined signature feature of road segment 5600. In oneexample, when count N_(c) matches n₂, processing unit may determinecurrent location 5628 as corresponding to location 2.

In some exemplary embodiments, the local features and predeterminedsignature features of road segment 5600 may be based on radii ofcurvature of the predetermined road model trajectory and an actualtrajectory travelled by vehicle 200. For example, as illustrated in FIG.59, vehicle 200 may travel over road segment 5900, which may includelanes 5902 and 5904. As illustrated in FIG. 59, lane 5902 may bedelimited by road center 5906 and right side 5908, whereas lane 5904 maybe delimited by left side 5910 and road center 5906. Vehicle 200 may beconfigured to travel along predetermined road model trajectory 5912,which may define a preferred path (e.g., a target road model trajectory)within lane 5902 of road segment 5900 that vehicle 200 may follow asvehicle 200 travels along road segment 5900. As also illustrated in FIG.59, vehicle 200 may travel via previous locations 5922, 5924, 5926,5928, 5930, 5932 to current location 5934. Although only six previouslocations 5922-5932 are illustrated in FIG. 59, one of ordinary skill inthe art would recognize that any number of previous locations of vehicle200 may be present on road segment 5900.

Processing unit 110 may determine travelled trajectory 5914 of vehicle200 as passing through previous locations 5922-5932 of vehicle 200. Inone exemplary embodiment, processing unit 110 may fit a curve, which maybe a three-dimensional polynomial similar to that representingpredetermined road model trajectory 5912, through locations 5922-5932.Processing unit 110 may also determine first parameter valuesrepresentative of curvatures of various segments (portions or sections)of predetermined road model trajectory 5912. Further, processing unit110 may determine second parameter values representative of a curvatureof travelled trajectory 5914. Processing unit 110 may determine currentlocation 5934 of vehicle 200 based on the first and second parametervalues.

For example, consider the case where R₁, R₂, R₃, . . . R_(z) representradii of curvature of segments C₁, C₂, C₃, . . . C_(z) of predeterminedroad model trajectory 5912. Referring to FIG. 59, portions C₁, C₂, C₃, .. . C_(z) of predetermined road model trajectory 5912 may representsections of predetermined road model trajectory 5912 between locations5922-5944, 5922-5946, 5922-5948, etc. Processing unit may determine, forexample, a radius of curvature R_(t) of travelled trajectory 5914between locations 5922 and 5934. Processing unit 110 may compare radiusof curvature R_(t) with the radii R₁, R₂, R₃, . . . R_(z). Processingunit 110 may determine current location 5934 of vehicle 200 as alocation 5970 when radius of curvature R_(t) matches the radius ofcurvature R_(p) of a portion of predetermined road model trajectorylying between location 5922 and 5970. Processing unit 110 may determinea match between radii R_(t) and R_(p) using matching techniques similarto those discussed above.

FIG. 60 is a flowchart showing an exemplary process 6000, for navigatingvehicle 200 along road segment 5900 (or 5600), using road signatures,consistent with disclosed embodiments. Steps of process 6000 may beperformed by one or more of processing unit 110 and image acquisitionunit 120, with or without the need to access memory 140 or 150. Theorder and arrangement of steps in process 6000 is provided for purposesof illustration. As will be appreciated from this disclosure,modifications may be made to process 6000 by, for example, adding,combining, removing, and/or rearranging the steps for the process.

As illustrated in FIG. 60, process 6000 may include a step 6002 ofreceiving information regarding one or more aspects of road segment 5900(or 5600) from a sensor. In one exemplary embodiment, sensor may includeone or more of image capture devices 122, 124, 126, which may acquireone or more images representative of an environment of the vehicle. Inone exemplary embodiment, image acquisition unit 120 may acquire one ormore images of an area forward of vehicle 200 (or to the sides or rearof a vehicle, for example). For example, image acquisition unit 120 mayobtain an image using image capture device 122 having a field of view202. In other exemplary embodiments, image acquisition unit 120 mayacquire images from one or more of image capture devices 122, 124, 126,having fields of view 202, 204, 206. Image acquisition unit 120 maytransmit the one or more images to processing unit 110 over a dataconnection (e.g., digital, wired, USB, wireless, Bluetooth, etc.).

In another exemplary embodiment, the sensor may include one or moresuspension sensors 5500 on vehicle 200. Suspension sensor 5500 may beconfigured to generate signals responsive to a movement of thesuspension of vehicle 200 relative to a surface of road segment 5900 (or5600). Processing unit 110 may receive signals from the one or moresuspension sensors 5500 on vehicle 200 as vehicle 200 moves along roadsegment 5900 (or 5600). For example, processing unit 110 may receiveinformation regarding the relative height of vehicle 200 adjacent eachof its wheels based on suspension sensors 5500 located adjacent to thewheels. Processing unit 110 may use this information to determine a roadsurface profile at the location of vehicle 200. The road surface profilemay provide information regarding a bank or inclination of, for example,lane 5902 (or 5602) relative to road center 5906 or right side 5908. Insome embodiments, the road surface profile may also identify a bump inroad segment 5900 (or 5600) based on the signals from the one or moresuspension sensors 5500.

Process 6000 may also include a step 6004 of determining a local featurebased on the information received from the sensor (e.g. imaging unit110, one or more suspension sensors 5500, etc.). The local feature mayrepresent one or more aspects of the road segment at current location5628 or 5932 of vehicle 200. For example, the local feature may includeat least one of a road width, a lane width, or a road surface profile atcurrent location 5932 of vehicle 200. In some exemplary embodiments, thelocal feature may be based on data collected by processing unit 110 asvehicle travels along predetermined road model trajectory to currentlocation 5932. In particular, based on road widths, lane widths, lengthsof road markings (dashes), spacing between adjacent road markings(dashes), etc., determined as vehicle travels to current location 5932,processing unit 110 may determine a road width profile, a lane widthprofile, a dashed line length profile, a dashed line spacing profile,second parameter values representing a curvature of travelled trajectory5914, and/or other parameters as discussed above with respect to FIGS.55-58.

Process 6000 may include a step 6006 of receiving predeterminedsignature features for road segment 5900. For example, processing unit110 may retrieve the predetermined signature features from a database160 stored in memories 140, 150 associated with vehicle 200 or from adatabase 160 located remotely from vehicle 200. As discussed above withrespect to FIGS. 56-59, the predetermined signature features may includeone or more predetermined parameter values representing at least one ofa road width, a lane width, a dashed line length, a dashed line spacing,etc., at predetermined locations along predetermined road modeltrajectory 5912. In some exemplary embodiments, the predeterminedsignature features may also include one or more of a road width profileover at least a portion of the road segment, a lane width profile overat least a portion of the road segment, a dashed line spacing profileover at least a portion of the road segment, a predetermined number ofroad markings along at least a portion of the road segment, a roadsurface profile over at least a portion of the road segment, or apredetermined curvature associated with the road segment at variouspredetermined locations along predetermined road model trajectory 5912.In some exemplary embodiments, processing unit 110 may retrieve a firstset of parameter values representing at least one predeterminedsignature feature of road segment 5900.

Further still, in some exemplary embodiments, the predeterminedsignature features may start at a known location (e.g., an intersection)and, if lane marking segment lengths and spacing are known and lanemarking segments are counted, processing unit 110 may determine alocation about the road from the known location. In some embodiments, acombination of known lengths for specific segments (e.g., typicallyclose to the intersection) together with statistics on regardingconsistent segment lengths and spacing may also be used as apredetermined signature feature. Further, in some embodiments, apredetermined signature feature may include a combination of tworepetitive features, such as the combination of lane marking segmentsand lampposts. In still yet other embodiments, a predetermined signaturefeature may include a combination of GPS data (e.g., an approximatelocation) and lane mark segments.

Process 6000 may also include a step 6008 of determining whether thelocal feature determined, for example, in step 6004 matches the at leastone predetermined signature feature, retrieved for example in step 6006.Processing unit 110 may determine whether there is a match as discussedabove with respect to FIGS. 57-59. When processing unit 110 determinesthat the local feature matches a predetermined signature feature (Step6008: Yes), processing unit 110 may proceed to step 6010. In step 6010,processing unit may determine current location 5628, 5932 of vehicle200. Processing unit 110 may determine current location 5932 asdiscussed above with respect to FIGS. 57-59. Returning to step 6008,when processing unit 110 determines, however, that the local featuredoes not match a predetermined signature feature (Step 6008: No),processing unit 110 may return to step 6006 to retrieve anotherpredetermined signature feature from database 160.

Process 6000 may include a step 6012 of determining heading direction5980 of vehicle 200 at current location 5628, 5932. Processing unit 110may determine heading direction 5980 using one or more operationsdiscussed above with respect to FIGS. 37-39. For example, processingunit 110 may determine heading direction 5980 as a gradient of travelledtrajectory 5914 at current location 5932 of vehicle 200. Process 6000may also include a step 6014 of determining a direction 5990 ofpredetermine road model trajectory 5912. Processing unit 110 maydetermine direction 5990 using one or more operations discussed abovewith respect to FIGS. 37-43. For example, processing unit 110 maydetermine direction 5990 as a vector oriented tangentially topredetermined road model trajectory 5912 at current location 5932 ofvehicle 200. Processing unit 110 may determine the tangential vector asa vector pointing along a gradient of the mathematical representation ofpredetermined road model trajectory 5912 at current location 5932.

Process 6000 may also include a step 6016 of determining an autonomoussteering action for vehicle 200. Processing unit 110 may determine arotational angle □ between heading direction 5980 and direction 5990 ofpredetermined road model trajectory 5912. Processing unit 110 mayexecute the instructions in navigational module 408 to determine anautonomous steering action for vehicle 200 that may help ensure thatheading direction 5980 of vehicle 200 is aligned (i.e., parallel) withdirection 5990 of predetermined road model trajectory 5912 at currentlocation 5932 of vehicle 200. Processing unit 110 may also send controlsignals to steering system 240 to adjust rotation of the wheels ofvehicle 200 to turn vehicle 200 so that heading direction 5980 may bealigned with direction 5990 of predetermined road model trajectory 5912at current location 5932.

Processing unit 110 and/or image acquisition unit 120 may repeat steps6002 through 6016 after a predetermined amount of time. In one exemplaryembodiment, the predetermined amount of time may range between about 0.5seconds to 1.5 seconds. By repeatedly determining current location 5932of vehicle 200 based on road signatures, heading direction 5980 based ontravelled trajectory 5914, direction 5990 of predetermined road modeltrajectory 3410 at current location 5932, and the autonomous steeringaction required to align heading direction 5980 with direction 5990,processing unit 110 and/or image acquisition unit 120 may help tonavigate vehicle 200, using road signatures, so that vehicle 200 maytravel along road segment 5912.

Forward Navigation Based on Rearward Facing Camera

Consistent with disclosed embodiments, in situations where adverselighting conditions inhibit navigation using a forward facing camera(e.g., driving into bright sun), navigation can be based on imageinformation obtained from a rearward facing camera.

In one embodiment, a system for autonomously navigating a vehicle mayinclude at least one processor. The at least one processor may beprogrammed to receive from a rearward facing camera, at least one imagerepresenting an area at a rear of the vehicle, analyze the at least onerearward facing image to locate in the image a representation of atleast one landmark, determine at least one indicator of position of thelandmark relative to the vehicle, determine a forward trajectory for thevehicle based, at least in part, upon the indicator of position of thelandmark relative to the vehicle, and cause the vehicle to navigatealong the determined forward trajectory.

In related embodiments, the indicator of position of the landmark mayinclude a distance between the vehicle and the landmark and/or arelative angle between the vehicle and the landmark. The landmark mayinclude a road marking, a lane marking, a reflector, a pole, a change inline pattern on a road, a road sign, or any other observable featureassociated with a road segment. The landmark may include a backside of aroad sign, for example. The at least one processor may be furtherprogrammed to determine a lane offset amount of the vehicle within acurrent lane of travel based on the indicator of position of thelandmark, and determination of the forward trajectory may be based onthe determined lane offset amount. The at least one processor may befurther programmed to receive from another camera, at least one imagerepresenting another area of the vehicle, and determination of theforward trajectory may be further based on the at least one imagereceived from the other camera.

In some embodiments, the rearward facing camera may be mounted on anobject connected to the vehicle. The object may be a trailer, a bikecarrier, a mounting base, a ski/snowboard carrier, or a luggage carrier.The rearward camera interface may be a detachable interface or awireless interface.

FIG. 61A is a diagrammatic side view representation of an exemplaryvehicle 6100 consistent with the disclosed embodiments. Vehicle 6100 maybe similar to vehicle 200 of FIG. 2A, except that vehicle 6100 includesin its body an image capture device 6102 facing in a rearward directionrelative to vehicle 6100. System 6104 may be similar to system 100 ofFIG. 1 and may include a processing unit 6106 similar to processing unit110. As shown in FIG. 61A, image capture device 6102 may be positionedin the vicinity of a trunk of vehicle 6100. Image capture device 6102may also be located, for example, at one of the following locations: onor in a side mirror of vehicle 6100; on the roof of vehicle 6100; on aside of vehicle 6100; mounted on, positioned behind, or positioned infront of any of the windows/windshield of vehicle 6100; on or in a rearbumper; mounted in or near light figures on the back of vehicle 6100; orany other locations where image capture device 6102 may capture an imageof an area rear of vehicle 6100. In some embodiments, as discussedabove, image capture device 6102 may be mounted behind a glare shieldthat is flush with the rear windshield of vehicle 6100. Such a shieldmay minimize the impact of reflections from inside vehicle 6100 on imagecapture device 6102.

FIG. 61A shows one image capture device 6102 facing a rearward directionof vehicle 6100. However, other embodiments may include a plurality ofimage capture devices located at different positions and facing arearward direction of vehicle 6100. For example, a first image capturedevice may be located in the trunk of vehicle 6100 facing a rearward,slightly downward direction of the vehicle 6100, and a second imagecapture device may be mounted on the roof of vehicle 6100 facing arearward, slightly upward direction of vehicle 6100. In another example,a first image capture device may be mounted on a left side mirror ofvehicle 6100, and a second image capture device may be mounted on aright side mirror of vehicle 6100. Both the first and second imagecapture devices may face a rearward direction of vehicle 6100.

In some embodiments, the relative positioning of the image capturedevices may be selected such that the fields of view of the imagecapture devices overlap fully, partially, or not at all. Further theimage capture devices may have the same or different fields of view andthe same or different focal lengths.

FIG. 61A shows an image capture device 6102 facing in a rearwarddirection relative to vehicle 6100. However, a skilled artesian wouldrecognize that vehicle 6100 may further include any number of imagecapture devices facing in various directions and that processing unit6106 may be further programmed to operate these additional image capturedevices. For example, vehicle 6100 may further include image capturedevices 122 and 124 of FIG. 2A, and processing unit 6106 may beprogrammed to perform the programmed functions of processing unit 110 ofsystem 100. A skilled artesian would further recognize that having aplurality of image capture devices, one facing a rearward direction ofvehicle 6100 and another facing a forward direction of vehicle 6100, maybe beneficial in situations where adverse lighting conditions mayinhibit navigation using one of the image capture devices (e.g., drivinginto bright sun). Since adverse lighting conditions rarely affect bothimage capture devices thereof, system 6104 may be configured to navigatebased on images received from an image capture device affected lessadversely by the lighting condition in these situations.

In some embodiments, a forward trajectory for the vehicle may bedetermined based on an image received from a rearward facing cameratogether with or independent from an image received from a forwardfacing camera. In some embodiments, a forward trajectory may bedetermined by, for example, averaging two determined trajectories, onebased on images received from a rearward facing camera, and anotherbased on images received from a forward facing camera.

In some embodiments, images from a forward facing camera and a rearwardfacing camera may be analyzed to determine which is currently providingmore useful images. Based on this determination, images from the forwardfacing camera or the rearward facing camera may selectively be used innavigating the vehicle. For example, in a situation where vehicle 6100may face a bright light source (e.g., the sun) that causes the forwardfacing camera to capture an image lacking sufficient detail on whichnavigational responses may accurately be determined, images collectedfrom a rearward facing camera, not affected by the same light source maybe used in navigating the vehicle. This determination and selection ofimages from the available image streams may be made on the fly.

In some embodiments, navigation may be based on images received from arearward facing camera because one or more objects (e.g., a large truckor other vehicle) is blocking a portion of a field of view of a forwardfacing camera. In other situations, navigation may be based on a imagescollected from a rearward facing camera as a supplement to imagescollected from a forward facing camera. For example, in some embodimentsa vehicle may locate a recognized landmark in a field of view of itsforward facing camera. From a time when that recognized landmark firstcomes into view of the forward facing camera until a time when thevehicle has passed the recognized landmark (or the landmark hasotherwise passed out of the field of view of the forward facing camera),navigation can proceed based on images captured of the recognizedlandmark (e.g., based on any of the techniques described above).Navigation based on the recognized landmark, however, need not end whenthe vehicle passes the landmark. Rather, a rearward facing camera cancapture images of the same recognized landmark as the vehicle travelsaway from the landmark. These images can be used, as described above, todetermine a location of the vehicle relative to a target trajectory fora particular road segment, and images of the backside of the recognizedlandmark may be usable as long as the backside of the landmark isvisible or appears in the images captured by the rearward facing camera.Using such a technique may extend an amount of time that a vehicle cannavigate with the benefit of a recognized landmark and delay a time whenthe vehicle must transition to dead reckoning or another navigationaltechnique not anchored by a known location of a recognized landmark. Asa result, navigational error may be even further reduced such that thevehicle even more closely follows a target trajectory.

In some embodiments, an object(s) may be present at the back of vehicle6100, and this object may be in the field of vision of image capturedevice 6102, interfering with image capture device's 6102 ability toaccurately capture images representing an area at a rear of vehicle6100. The object may be, for example, a trailer, a mounting base, a bikecarrier, a ski/snowboard carrier, or luggage carrier. In theseembodiments, image capture device 6102 may be mounted on the object andarranged to capture images representing an area at a rear of the object.FIG. 61B illustrates an exemplary vehicle with such an object.

FIG. 61B is a diagrammatic side view representation of an exemplaryvehicle 6150 consistent with the disclosed embodiments. Vehicle 6150 issimilar to vehicle 6100 except that vehicle 6150 is towing a trailer6108 and image capture device 6102 is mounted on trailer 6108. As shownin FIG. 61B, image capture device 6102 is facing a rearward direction oftrailer 6108 and positioned to capture images representing an area at arear of trailer 6108. As discussed above, the presence of trailer 6108may interfere with any image capture devices that may be mounted on thebody of vehicle 6150 and face a rearward direction of vehicle 6108.

In some embodiments, image capture device 6102 of FIG. 61B may have beenpreviously mounted in or on the body of vehicle 6150 (similar to imagecapture device 6102 of FIG. 61A) and is now repositioned on trailer6108. In these embodiments, image capture device 6102 may beelectrically connected to system 6104 via a detachable electricalinterface 6110. A “detachable electrical interface” may be broadlydefined as a set of connectors that can be connected and disconnected bya driver or a passenger (or any other person, who may not be a skilledartesian). Detachable electrical interface 6110 allows a user who maynot be a skilled artesian, such as the driver or passengers of vehicle6150, to remount image capture device 6102, for example, from trailer6108 to vehicle 6150. This capability may be especially useful insituations where vehicle 6150 frequently switches between operating withand without trailer 6108. In other embodiments, rearward facing camera6102 may be configured to communicate wirelessly with processing unit6106.

FIG. 62 is a diagrammatic top view representation of an exemplaryvehicle autonomously navigating on a road consistent with disclosedembodiments. FIG. 62 shows vehicle 6100 of FIG. 61A including imagecapture device 6102 (with its line of sight 6102A) and system 6104 forautonomously navigating vehicle 6100. FIG. 62 also shows severalpotential, recognized landmarks, including a road roughness profile 6208associated with a particular road segment, a change in lane markings6210, reflectors 6202A-C, a road sign 6204 facing away from vehicle6100, a road sign 6206 facing towards vehicle 6100, and a pole 6214.

FIG. 62 also shows indicators of position of a landmark relative tovehicle 6100. The indicators of position in FIG. 62 include a distance6212A and/or a relative angle 6212B between a landmark (e.g., reflector6202A) and vehicle 6100. An “indicator of position” may refer to anyinformation that relates to a position. Thus, an indicator of positionof a landmark may include any information related to the position of thelandmark. In the example of FIG. 62, indicators of position of alandmark are determined relative to vehicle 6100.

As shown in FIG. 62, distance 6212A is the distance between imagecapture device 6102 and the landmark, and angle 6212B is the anglebetween line of sight 6102A of image capture device 6102 and animaginary line from image capture device 6102 to the landmark. However,in some embodiments, distance 6212A may be the distance between areference point in the vicinity of vehicle 6100 and the landmark, andangle 6212B may be the angle between a reference line through thereference point and an imaginary line from the reference point to thelandmark. The reference point may be, for example, the center of vehicle6100, and the reference line may be, for example, a line through thecenter of vehicle 6100.

In some embodiments, one or more landmarks and/or one or more indicatorsof position may be used in autonomous navigation of vehicle 6100. Forexample, the indicators of position may be used to determine a currentlocation of a vehicle relative to a target trajectory stored in sparsemap 800, for example. Any of the techniques discussed above with respectto landmark recognition and use in determining one or more navigationalresponses for a vehicle may be employed based on images received from arearward facing camera.

FIG. 63 is a flowchart showing an exemplary process for using one ormore landmarks and one or more indicators of position for autonomouslynavigating a vehicle. Process 6300 may use at least one image from arearward facing camera and analyze the at least one image to navigate avehicle along a forward trajectory.

At step 6302, processing unit 6106 may receive from a rearward facingcamera, at least one image representing an area at a rear of vehicle6100. At an optional step, processing unit 6106 may receive from anothercamera, at least one image representing another area of vehicle 6100. Insome embodiments, processing unit 6106 may receive the images via one ormore camera interfaces. For example, processing unit 6106 may receive atleast one image representing an area at a rear of the vehicle via arearward camera interface and receive one image representing an area ata front of the vehicle via a forward camera interface. In someembodiments, as discussed above, the one or more camera interfaces mayinclude a detachable interface or a wireless interface.

At step 6304, processing unit 6106 may analyze the at least one rearwardfacing image to locate in the image a representation of at least onelandmark. As discussed above in reference to FIG. 62, a landmark mayinclude, for example, a road profile 6208, a change in lane markings6210, reflectors 6202A-C, a road sign 6204 facing away from vehicle6100, a road sign 6206 facing towards vehicle 6100, and a pole 621.Alternatively, or additionally, a landmark may include, for example, atraffic sign, an arrow marking, a lane marking, a traffic light, a stopline, a directional sign, a landmark beacon, a lamppost, a directionalsign, a speed limit sign, a road marking, a business sign, a distancemarker, or a change in spacing of lines on the road.

In some embodiments, before or after a landmark is located in thereceived image, processing unit 6106 may retrieve information relatingto recognized landmarks in the vicinity of the autonomous vehicle. Theinformation relating to landmarks may include, for example, informationrelating to size and/or shape of a landmark. The information relating tolandmarks may be retrieved from, for example, a database, which may belocated in system 6104 or located external to vehicle 6100 (connected tosystem 6104 via a communication system such as a cellular network orother wireless platform).

At step 6306, processing unit 6106 may determine at least one indicatorof position of the landmark relative to the vehicle. For example, anindicator of position may include a distance between a landmark and thevehicle and/or a relative angle between a landmark and the vehicle. Asdiscussed above in reference to FIG. 62, the distance may be, forexample, distance 6212A, which is the distance between image capturedevice 6102 and the landmark, and the angle may be angle 6212B, which isthe angle between line of sight 6102A of image capture device 6102 andan imaginary line from image capture device 6102 to the landmark.

In some embodiments, processing unit 6106 may determine at least oneindicator of position of the landmark relative to the vehicle based onthe representation of the located landmarks in the received image. Forexample, the size and shape of the representation of the locatedlandmarks in the received image may be used to estimate distance 6212Afrom vehicle 6100 (e.g., by monitoring a change in scale of the objectover multiple image frames). In another example, coordinates of thepixels occupied by the representation of the located landmarks in thereceived image may be used to estimate angle 6212B. In embodiments whereinformation relating to landmarks is retrieved by processing unit 6106,the information may be used to model and compare with the representationof the located landmarks in the received image. In these embodiments,the information may improve the accuracy of the determined indicators ofposition.

At an optional step 6308, processing unit 6106 may determine a laneoffset amount of the vehicle within a current lane of travel (or evenmake a determination of a current lane of travel) based on the indicatorof position of the landmark relative to the vehicle. For example, suchan offset determination or lane determination may be determined byknowing a position of a recognized landmark along with a relativepositioning of the recognized landmark with respect to lanes of a roadsegment. Thus, once a distance and direction are determined relative toa recognized landmark, the current lane of travel and/or amount of laneoffset within a particular lane of travel may be calculated. A laneoffset of a vehicle may refer to a perpendicular distance from a laneindicator to a reference point. In some embodiments, the reference pointmay correspond to an edge of a vehicle or a point along a centerline ofa vehicle. In other embodiments, the reference point may correspond to amid-point of a lane or a road. A lane indicator may include, forexample, a lane marking, a road edge, reflectors for improvingvisibility of a lane, or any other object that is on or near theboundaries of a lane. In the above example, the lane offset may be theperpendicular distance from road edge 6208 to vehicle 6100.

At step 6310, processing unit 6106 may determine a forward trajectoryfor the vehicle based, at least in part, upon the indicator of positionof the landmark relative to the vehicle. In embodiments where theoptional step 6308 is performed, processing unit 6106 may determine theforward trajectory for the vehicle further based on the determined laneoffset amount. In embodiments where processing unit 6106 received fromanother camera, at least one image representing another area of vehicle6100, processing unit 6106 may determine the forward trajectory for thevehicle further based on the at least one image received from theanother camera. Such a trajectory determination may be based on any ofthe techniques described above (e.g., navigation based on recognizedlandmarks, tail alignment, etc.)

At step 6312, processing unit 6106 may cause vehicle 6100 to navigatealong the determined forward trajectory. In some embodiments, processingunit 6106 may cause one or more navigational responses in vehicle 6100to navigate along the determined forward trajectory. Navigationalresponses may include, for example, a turn, a lane shift, a change inacceleration, and the like. Additionally, multiple navigationalresponses may occur simultaneously, in sequence, or any combinationthereof to navigate along the determined forward trajectory. Forinstance, processing unit 6106 may cause vehicle 6100 to shift one laneover and then accelerate by, for example, sequentially transmittingcontrol signals to steering system 240 and throttling system 220.Alternatively, processing unit 110 may cause vehicle 6100 to brake whileat the same time shifting lanes by, for example, simultaneouslytransmitting control signals to braking system 230 and steering system240.

Navigation Based on Free Space Determination

Consistent with disclosed embodiments, the system can recognize parkedvehicles, road edges, barriers, pedestrians, and other objects todetermine free space boundaries within which the vehicle can travel.

In some situations, free space boundaries may be used to navigate avehicle. These situations may include, for example, when lane markingsare not visible because lanes do not exist and/or because obstacles arecovering the lane marks (e.g., parked cars and snow). Alternatively,free space boundaries may be used to navigate a vehicle in addition tothe lane-mark-based navigation method to increase the robustness of thesystem.

FIG. 64 is a diagrammatic perspective view of an environment 6400captured by a forward facing image capture device on an exemplaryvehicle consistent with disclosed embodiments. The exemplary vehicle maybe, for example, vehicle 200 described above in reference to FIGS. 2A-2Fand may include a processing unit, such as processing unit 110 ofvehicle 200. The forward facing image capture device may be, forexample, image capture device 122, image capture device 124, or imagecapture device 126 of vehicle 200.

FIG. 64 shows a non-road area 6410 with a road edge 6410A, a sidewalk6412 with a curb 6412A, and a road horizon 6414. A road barrier 6418 maybe present in the vicinity of road edge 6410A, and a car 6416 may beparked in the vicinity of curb 6412A. FIG. 64 also shows a first freespace boundary 6404, a second free space boundary 6406, and a forwardfree space boundary 6408. Forward free space boundary 6408 may extendbetween first free space boundary 6404 and second free space boundary6406. A free space region 6402 forward of the vehicle (not shown in thefigure) may be a region bound by these three boundaries and mayrepresent a physically drivable region within environment 6400. Firstand second free space boundaries 6404, 6406 may each correspond to, forexample, a road edge, a road barrier, a parked car, a curb, a lanedividing structure, a tunnel wall, and/or a bridge structure, or acombination thereof. Forward free space 6408 may correspond to, forexample, an end of the road, a road horizon, a road barrier, a vehicle,or a combination thereof.

In the example of FIG. 64, first and second free space boundaries 6404,6406 may each correspond to a plurality of objects. For example, firstfree space boundary 6404 may correspond to a portion of road edge 6410Aand road barrier 6418, and second free space boundary 6406 maycorrespond to a portion of curb 6412A and parked car 6414. However, insome embodiments, each free space boundary may correspond to a singleobject. Similarly, forward free space 6408 may correspond to one or moreobjects. For example, in FIG. 64, forward free space 6408 corresponds toroad horizon 6414.

In some embodiments, one or more obstacles may exist in free spaceregion 6402 bound by the three free space boundaries 6404, 6406, and6408. In these embodiments, the obstacles may be excluded from freespace region 6402. In FIG. 64, for example, pedestrian 6420 is standinginside free space region 6402 bound by the three free space boundaries6404, 6406, and 6408. Therefore, pedestrian 6420 may be excluded fromfree space region 6402. Alternatively, regions surrounding the obstaclesmay be excluded from free space region 6402. For example, a region 6422surrounding pedestrian 6420, instead of the region occupied bypedestrian 6420, may be excluded from free space region 6402. Obstaclesmay include, for example, a pedestrian, another vehicle, and debris.

A size of a region (e.g., region 6422) surrounding an obstacle (e.g.,pedestrian 6420) may determine the minimum distance that may existbetween the vehicle and the obstacle during navigation. In someembodiments, the size of the region may be substantially the same as thesize of the obstacle. In other embodiments, the size of the region maybe determined based on the type of the obstacle. For example, region6422 surrounding pedestrian 6420 may be relatively large for safetyreasons, while another region surrounding debris may be relativelysmall. In some embodiments, the size of the region may be determinedbased on the speed at which the obstacle is moving, frame rate of theimage capture device, speed of the vehicle, or a combination thereof. Insome embodiments, a shape of a region surrounding an obstacle may be acircle, a triangle, a rectangle, or a polygon.

In FIG. 64, first free space boundary 6404 corresponds to a portion ofroad edge 6410 and road barrier 6148, and second free space boundary6406 corresponds to a portion of curb 6412A and parked car 6416.However, in other embodiments, first and second free space boundaries6404, 6406 may correspond to road edge 6410A and curb 6412A,respectively, and road barrier 6148 and parked car 6416 may beconsidered as obstacles.

In some embodiments, regions between obstacles may be excluded from freespace region 6402. For example, if a width of a region between twoobstacles is less than the width of the vehicle, the region may beexcluded from free space region 6402.

FIG. 65 is an exemplary image received from a forward facing imagecapture device of a vehicle consistent with disclosed embodiments. FIG.65 shows a first free space boundary 6504 and a second free spaceboundary 6506. Both free space boundaries correspond to curbs and parkedcars (e.g., parked car 6516) on each side of the road. FIG. 65 alsoshows a free space region 6502, which may be defined by first free spaceboundary 6504, a second free space boundary 6506, and a forward freespace boundary (not shown). Additionally, FIG. 65 shows a moving car6520, which may be considered as an obstacle. Therefore, moving car 6520may be excluded from free space region 6502.

FIG. 66 is a flowchart showing exemplary process 6600 for navigatingvehicle 200 based on free space region 6402 in which vehicle 200 cantravel consistent with disclosed embodiments. Process 6300 may use aplurality of images from a forward facing image capture device, analyzeat least one image of the plurality of images to identify free spaceboundaries and define a free space region bound by the identified freespace boundaries. Furthermore, process 6300 may navigate a vehicle basedon the defined free space region.

At step 6602, processing unit 110 may receive from image capture device122, a plurality of images associated with environment 6400 of vehicle200. As discussed above, FIG. 65 is an example of an image that may bereceived from image capture device 122. In some embodiments, images maybe captured at different times by image capture device 122 (e.g., imagesmay be captured apart by less than a second, 1 second, 2 second, etc.).In some embodiments, vehicle 200 may include a plurality of imagecapture devices (e.g., image capture devices 122 and 124 of vehicle200), and processing unit 110 may receive from each image capturedevice, a plurality of images associated with environment 6400 ofvehicle 200. The plurality of images received from each image capturedevice may be images captured at different times by each image capturedevice.

At step 6604, processing unit 110 may analyze at least one of theplurality of images received from, for example, image capture device122. In embodiments where a single plurality of images is generatedbased on images received from a plurality of image capture devices,processing unit 110 may analyze at least one image of the singleplurality of images. Alternatively, each image received from each imagecapture device may be analyzed independently.

Additionally, processing unit 110 may identify a first free spaceboundary 6404 on a driver side of vehicle 200 and extending forward ofvehicle 200, a second free space boundary 6406 on a passenger side ofvehicle 200 and extending forward of vehicle 200, and a forward freespace boundary 6408 forward of vehicle 200 and extending between firstfree space boundary 6404 and second free space boundary 6406.Additionally, processing unit 110 may further identify a free spaceregion 6402 forward of the vehicle as the region bound by first freespace boundary 6404, the second free space boundary 6406, and forwardfree space boundary 6408. As discussed above, first and second freespace boundaries 6404, 6406 may each correspond to, for example, a roadedge, a road barrier, a parked car, a curb, a lane dividing structure, atunnel wall, and/or a bridge structure, or a combination thereof.Furthermore, as discussed above, forward free space 6408 may correspondto, for example, an end of the road, a road horizon, a road barrier, avehicle, or a combination thereof.

At an optional step, processing unit 110 may identify, based on theanalysis at step 6604, an obstacle (e.g., pedestrian 6420) forward ofvehicle 200 and exclude the identified obstacle from free space region6402 forward of vehicle 200. Alternatively, processing unit 110 mayidentify, based on analysis of the at least one of the plurality ofimages at step 6640, an obstacle (e.g., pedestrian 6420) forward of thevehicle and exclude a region (e.g., region 6422) surrounding theidentified obstacle from free space region 6402 forward of the vehicle.

In some embodiments, as discussed above, the size of the regionsurrounding the identified obstacle may be substantially the same as thesize of the obstacle, or alternatively, the size of the regionsurrounding the obstacle may be determined based on the type ofobstacle. In some embodiments, as discussed above, the size of theregion may be determined based on the speed at which the obstacle ismoving, a frame rate of image capture device 122, speed of vehicle 200,or a combination thereof.

At another optional step, processing unit 110 may exclude, from freespace region 6402, regions between the identified obstacles and/orregions between identified obstacles and free space boundaries 6404,6406. In some embodiments, as discussed above, processing unit 110 maydetermine whether to exclude the regions between the identifiedobstacles based on a distance between the identified obstacles.Furthermore, processing unit 110 may determine whether to exclude theregions between identified obstacles and free space boundaries 6404,6406 based on the distance between identified obstacles and free spaceboundaries 6404, 6406.

At step 6606, processing unit 110 may determine a navigational path forvehicle 200 through free space region 6402. In some embodiments, thenavigational path may be a path through the center of free space region6402. In other embodiments, the navigational path may be a path that isa predetermined distance away from one of first and second free spaceboundaries 6404, 6406. The predetermined distance may be a fixeddistance, or alternatively, the predetermined distance may be determinedbased on, for example, one or more of the following: speed of vehicle200, width of free space region 6402, and number of obstacles withinfree space region 6402. Alternatively, the navigational path may be, forexample, a path that uses the minimum number of navigational responsesor the shorted path.

At step 6608, processing unit 110 may cause vehicle 200 to travel on atleast a portion of the determined navigational path within the freespace region 6402 forward of vehicle 200. In some embodiments,processing unit 110 may cause one or more navigational responses invehicle 200 to navigate along the determined navigational path.Navigational responses may include, for example, a turn, a lane shift, achange in acceleration, and the like. Additionally, multiplenavigational responses may occur simultaneously, in sequence, or anycombination thereof to navigate along the determined forward trajectory.For instance, processing unit 110 may cause vehicle 200 to movelaterally and then accelerate by, for example, sequentially transmittingcontrol signals to steering system 240 and throttling system 220.Alternatively, processing unit 110 may cause vehicle 200 to brake whileat the same time moving laterally, for example, simultaneouslytransmitting control signals to braking system 230 and steering system240. Further, for example, the free space boundary may serve as alocalization aid in the same way that lane marks are being used. Oncethe free space boundary is encoded, it may describe a 3D curve in space.At the localization stage, the projection of that 3D curve to the imagemay provide a localization cue, since it may collide with the free spacedetection at that location.

Navigating in Snow

Consistent with disclosed embodiments, the system may determine theedges of a road in poor weather conditions, such as when a road iscovered in snow. For example, the system may take into account changesin light, the curve of a tree line, and tire tracks to determineprobable locations of the edges of the road.

FIG. 67 is a diagrammatic top view representation of an exemplaryvehicle navigating on a road with snow covering at least some lanemarkings and road edges consistent with disclosed embodiments. Theexemplary vehicle may be, for example, vehicle 200 described above inreference to FIGS. 2A-2F and may include a processing unit, such asprocessing unit 110 of vehicle 200. The forward facing image capturedevice may be, for example, image capture device 122, image capturedevice 124, or image capture device 126 of vehicle 200.

In FIG. 67, the road may include a driver side lane mark 6702 and apassenger side lane mark 6704. FIG. 67 also shows a non-road area 6710and a road edge 6706. In one example, non-road area 6710 may be anon-paved area, a sidewalk, or a beginning of a hill. In anotherexample, non-road area 6710 may be an area without a platform, such asan area with a sharp vertical drop (i.e., cliff).

FIG. 67 also shows an area covered by snow 6708. Specifically, area 6708covers a portion of road edge 6706 and portions of lane marks 6702,6704. Thus, road edge 6706 and/or one or more of lane marks 6702, 6704may not be readily apparent through analysis of images captured duringnavigation of vehicle 200. In such situations, vehicle 200 may navigatebased on analysis of captured images by determining probable locationsfor road edges bounding the portion of the road that is covered withsnow.

In some embodiments, the determination of the probable locations forroad edges may be based on tire tracks 6712 over an area covered by snow6708. For example, the presence of tire tracks 6712 may indicate thatthe portion of an area covered by snow 6708 with tire tracks 6712 iswithin the bounds of road edges. In some embodiments, the processingunit of the vehicle may consider the path of the tire tracks as a viablenavigational path and may cause the vehicle to follow the tire trackssubject to consideration of other criteria (e.g., whether the tracksremain within an area determined as likely corresponding to the road or,more specifically, a lane of travel for the vehicle).

In other embodiments, the determination of the probable locations forroad edges may be based on a change of light across a surface of areacovered by snow 6708. The source of the light may include, for example,headlights of vehicle 200, light from other vehicles, street lights, orthe sun. The change of light across the surface of area covered by snow6708 may occur for various reasons. In one example, surface roughness ofnon-road area 6710 and surface roughness of the road may be different;non-road area 6710 may be a gravel area, while the road may be a pavedarea. In another example, non-road area 6710 and the road may not belevel. Non-road area 6710 may be a sidewalk, which is typically raisedabove the road; alternatively, non-road area 6710 may be a hill or acliff Each of these may alter the surface of a covering of snow and maybe recognized based on certain variations in the surface of the snowcovering (e.g., changes in height, changes in texture, etc.) which maybe accentuated by shadows cast across the snow surface.

In other embodiments, the determination of the probable locations forroad edges may be based on a plurality of trees (e.g., forming a treeline) along an edge of the road. For example, in FIG. 67, trees 6714 maybe present along the road edge and visible even when snow covers roadedge 6706. In situations where trees 6714 are present close to road edge6714, the location of trees 6714 may be used as a probable location of aroad edge. However, in some situations, trees 6714 may be present somedistance away from a road edge. Therefore, in some embodiments, theprobable location of a road edge may be determined as a location that isa distance away from the location of trees 6714. The distance may be afixed value, or alternatively, the distance may be dynamicallydetermined based on, for example, a last visible road edge.

In other embodiments, the determination of the probable locations forroad edges may be based on an observed changes in curvature at a surfaceof the snow. The change in curvature at a surface of the snow may occurfor various reasons. For example, a change in curvature at a surface ofsnow may occur when non-road area 6710 and the road are not level. Insituations where non-road area 6710 is, for example, a sidewalktypically raised above the road, the snow may pile up near road edge6706 thereby changing the curvature of snow near road edge 6706. Inother situations, the non-road area may be a beginning of a hill, andthe curvature at the surface of the snow may follow the curvature of thehill beginning at road edge 6706. In these embodiments, the locationwhere the curvature begins to change may be determined as a probablelocation of a road edge.

FIG. 68 is a flowchart showing an exemplary process 6800 for navigatingvehicle 200 on a road with snow covering at least some lane markings androad edges consistent with disclosed embodiments. Process 6800 may usethe probable locations for road edges, as described above, to navigatevehicle 200.

At 6802, processing unit 110 may receive from an image capture device,at least one environmental image forward of the vehicle, including areaswhere snow covers at least some lane markings (e.g., lane marks 6702,6704) and road edges (e.g., road edge 6706).

At step 6804, processing unit 110 may identify, based on an analysis ofthe at least one image, at least a portion of the road that is coveredwith snow and probable locations for road edges bounding the at least aportion of the road that is covered with snow. As discussed above, theanalysis of the at least one image may include identifying at least onetire track in the snow, a change of light across a surface of the snow,and/or a trees along an edge of the road. Further, as discussed above,the analysis of the at least one image may include recognizing a changein curvature at a surface of the snow, where the recognized change incurvature is determined to correspond to a probable location of a roadedge.

In some embodiments, the analysis of the at least one image may includea pixel analysis of the at least one image in which at least a firstpixel is compared to at least a second pixel in order to determine afeature associated with a surface of the snow covering at least somelane markings and road edges. For example, each pixel in the image maybe compared with every adjacent pixel. In some embodiments, a color ofthe first pixel may be compared to a color of at least the second pixel.Alternatively, or additionally, an intensity of a color component of thefirst pixel may be compared to an intensity of the color component of atleast the second pixel. In other embodiments, the following propertiesof a pixel may be compared.

In some embodiments, the pixel analysis may identify features such as anedge of tire track 6712 or road edge 6706. The analysis for identifyingsuch features may include identifying a set of pixels where a rate inwhich a pixel property changes exceeds a threshold rate. The pixelproperty may include, for example, color of a pixel and/or intensity ofa color component of a pixel.

At step 6806, processing unit 110 may cause the vehicle to navigate anavigational path that includes the identified portion of the road andfalls within the determined probable locations for the road edges.

In embodiments where the probable location for road edges are determinedbased on the identified tire tracks 6712, processing unit 110 may causevehicle 200 to navigate by at least partially following the identifiedtire tracks 6712 in the snow. In embodiments where the probable locationfor road edges (e.g., road edges 6702, 6704) are determined based on achange of light across a surface of area covered by snow 6708, aplurality of trees (e.g., forming a tree line) along an edge of theroad, and/or a change in curvature at a surface of the snow, processingunit 110 may cause vehicle 200 to navigate between the determined edgesof the road.

Furthermore, in embodiments where edges of the road are determined byanalyzing pixels of the image received at step 6802, processing unit 110may cause vehicle 200 to navigate between the determined edges of theroad. In embodiments where an edge of a tire track is determined byanalyzing pixels of the image received at step 6802, processing unit 110may cause vehicle 200 to navigate by at least partially following tiretracks in the snow.

In some embodiments, processing unit 110 may cause one or morenavigational responses in vehicle 200 to navigate along the determinednavigational path. Navigational responses may include, for example, aturn, a lane shift, a change in acceleration, and the like.Additionally, multiple navigational responses may occur simultaneously,in sequence, or any combination thereof to navigate along the determinedforward trajectory. For instance, processing unit 110 may cause vehicle200 to move laterally and then accelerate by, for example, sequentiallytransmitting control signals to steering system 240 and throttlingsystem 220. Alternatively, processing unit 110 may cause vehicle 200 tobrake while at the same time moving laterally, for example,simultaneously transmitting control signals to braking system 230 andsteering system 240.

Additional techniques may also be employed by processing unit 110 fornavigating a vehicle on a road at least partially covered with snow. Forexample, in some embodiments, one or more neural networks may beemployed to aid in determination of a proposed path of travel along aroad covered in snow. This technique may be referred to as holistic pathprediction (HPP). Such a neural network may be trained, for example, bybeing supplied with images as a user drives along a road. To train theneural network in navigation of a snow covered road, various testingsituations involving snow covered roads may be used. Using images(perhaps thousands of training images, millions of images, or more) ofroads covered with snow captured as a driver navigates a vehicle alongsnow covered roads, the neural network will learn to develop a proposednavigational path along the snow. The process may involve setting up theneural network to periodically or continuously generate a proposednavigational path based on observed features of the snow covered road(including, for example, aspects of the surface of the road, edges ofthe road, sides of the road, barriers present, objects adjacent to theroad, cars on the road, etc.) and test the proposed navigational pathagainst actual behavior of the driver. Where the proposed navigationalpath diverges from the actual path the driver follows, the neuralnetwork will analyze the available images and make adjustments to itsprocessing algorithm in order to provide a different response in asimilar situation in the future (e.g., to provide a proposednavigational path that more closely matches the behavior of the driver).Once trained, the neural network may provide a proposed navigationalpath over a road covered with snow. Navigation through snow may be basedsolely on the output of a single trained neural network

In some embodiments, however, other techniques may be used to navigatethe vehicle through snow. In some embodiments, the free spacedetermination technique described in another section of this disclosuremay be used to define a path forward of the vehicle through an areaperceived as free space. For example, based on a captured image or imagestream, processing unit 110 may analyze at least one of the plurality ofimages to identify a first free space boundary on a driver side of thevehicle and extending forward of the vehicle. A second free spaceboundary may be identified on a passenger side of the vehicle andextending forward of the vehicle. A forward free space boundary may beidentified forward of the vehicle and extending between the first freespace boundary and the second free space boundary. Of course, theseboundaries need not be straight lines, but instead, can be representedby a complex series of curves or line segments that delineate sometimeshighly irregular boundary conditions (especially on the sides of thevehicle). Together, first free space boundary, the second free spaceboundary, and the forward free space boundary define a free space regionforward of the vehicle. Processing unit 110 may then determine aproposed navigational path for the vehicle through the free spaceregion. Navigation of the vehicle through snow may be based on the freespace determination technique alone. It should be noted that the freespace determination technique may be implemented using one or moreneural networks. In some embodiments, the neural network that implementsthe free space determination technique may be different from the neuralnetwork that implements the HPP technique.

In some embodiments, navigation through snow may be based on one or moretechniques used in combination. For example, any of the disclosednavigational systems may be used together to navigate a vehicle in snow.In some embodiments, the free space determination technique may becombined with the HPP technique. That is, a plurality of captured imagesmay be supplied to a neural network implementing the free spacetechnique in order to obtain a first proposed navigational path for thevehicle. The plurality of captured images may also be supplied to theneural network implementing the HPP technique to obtain a secondproposed navigational path for the vehicle. If the processing unitdetermines that the first proposed navigational path agrees with thesecond proposed navigational path, then the processing unit may causethe vehicle to travel on at least a portion of one of the proposednavigational paths (or an aggregate of the proposed navigational paths).In this context, agreement does not necessarily require an exact matchof the proposed navigational paths. Rather, agreement may be determinedif the proposed navigational paths have greater than a predetermineddegree of correlation (which may be determined using any suitablecompare function).

If the first proposed navigational path does not agree with the secondproposed navigational path, then a prompt may be provided to the user totake over control of at least some aspect of the vehicle. Alternatively,additional information may be considered in order to determine anappropriate navigational path for the vehicle. For example, where thereis disagreement in the proposed navigational paths from the free spaceand HPP techniques, processing unit 110 may look to a target trajectoryfrom sparse data map 800 (along with an ego motion estimation orlandmark based determination of a current position relative to thetarget trajectory) to determine a direction of travel for the vehicle.Outputs from other modules operating on processing unit 110 may also beconsulted. For example, a vehicle detection module may provide anindication of the presence of other vehicles in the environment of thehost vehicle. Such vehicles may be used to aid in path prediction forthe host vehicle (e.g., by following a lead vehicle, avoiding a parkedvehicle, etc.). A hazard detection module may be consulted to determinethe presence of any edges in or along the roadway having a heightexceeding a threshold. A curve detection module may be consulted tolocate a curve forward of the vehicle and to propose a path through thecurve. Any other suitable detection/analysis module operating onprocessing unit 110 may also be consulted for input that may aid inestablishing a valid path forward for the host vehicle.

The description and the figures above show a road that is covered bysnow; however, in some embodiments, a road may be covered with object(s)other than snow. For example, the road may be covered with sand orgravel instead of snow, and the disclosed embodiments may similarly beapplied to roads covered with these objects.

Autonomous Vehicle Speed Calibration

In some situations, vehicle navigation can be based on dead reckoning(for example, at least for short segments) where the vehicle determinesits current location based on its last known position, its speedhistory, and its motion history. Dead reckoning, however, may introduceaccumulating errors because every new position determination may relyupon measurements of translational and rotational velocities, which mayintroduce a certain level of error. Similarly, each new positiondetermination may rely upon a previously determined coordinate, which,in turn, may have been based on measurements including their owninaccuracies. Such inaccuracies and errors may be imparted into the deadreckoned position determinations through various sources, such as theoutputs of vehicle speed sensors for example. Even small inaccuracies inspeed sensing may accumulate over time. For example, in some cases,small errors in speed sensing (e.g., on the order of 1 km/hr or evenless) may result position determination errors on the order of 1 meter,5 meters, or more over a kilometer. Such errors, however, may be reducedor eliminated through calibration of vehicle speed sensors. According tothe disclosed embodiments, such calibration may be performed by anautonomous vehicle based on known landmark positions or based on areference distance along a road segment being traversed by the vehicle.

FIG. 69 is a diagrammatic top view representation of an exemplaryvehicle including a system for calibrating a speed of the vehicleconsistent with disclosed embodiments. The exemplary vehicle may be, forexample, vehicle 200 described above in reference to FIGS. 2A-2F and mayinclude a processing unit, such as processing unit 110 of vehicle 200.The forward facing image capture device may include, for example, imagecapture device 122, image capture device 124, or image capture device126 of vehicle 200. Such image capture devices may be configured toobtain images of an environment forward, to the side, and/or to the rearof vehicle 200.

In some embodiments, vehicle 200 may include various sensors. Suchsensors may include one or more speed sensors, GPS receivers,accelerometers, etc.

In some embodiments, recognized landmarks may be used in a speedcalibration process for the vehicle. Such recognized landmarks mayinclude those landmarks represented in sparse map 800, for example. FIG.69 shows examples of landmarks that may be used for calibrating speed ofvehicle 200. For example, FIG. 69 shows landmarks such as a traffic sign6906, a dashed lane marking 6902, a traffic light 6908, a stop line6912, reflectors 6910, and a lamp post 6914. Other landmarks mayinclude, for example, an arrow marking, a directional sign, a landmarkbeacon, a speed bump 6904, etc.

In some embodiments, processing unit 110 of vehicle 200 may identify oneor more recognized landmarks. Processing unit 110 may identify the oneor more recognized visual landmarks based on any of the previouslydescribed techniques. For example, processing unit 110 may receive alocal map associated with sparse map 800 (or may even receive or beloaded with sparse map 800) including representations of recognizedlandmarks. Because these landmarks may be indexed and/or becauseprocessing unit 110 may be aware of a current position of vehicle 200(e.g., with respect to a target trajectory along a road segment),processor unit 110 may anticipate a location for the next expectedrecognized landmark as it traverses a road segment. In this way,processor unit 110 may even “look” to a particular location withinimages received from image capture device 122 where the next recognizedlandmark is expected to appear. Once the recognized landmark is locatedwithin a captured image or captured images, processor unit 110 mayverify that the landmark appearing in the images is the expectedrecognized landmark. For example, various characteristics associatedwith the landmark in a captured image may be compared with informationstored in sparse data map 800 relative to the recognized landmark. Suchcharacteristics may include a size, landmark type (e.g., speed limitsign, hazard sign, etc.), position, distance from a previous landmark,etc. If the observed characteristics for a landmark match those storedrelative to a recognized landmark, then processor unit 110 can concludethat the observed landmark is the expected recognized landmark.

In some embodiments, after identifying a recognized landmark, processingunit 110 may retrieve information associated with the recognizedlandmarks. The information may include, for example, positionalinformation of the recognized landmarks. In some embodiments, theinformation associated with the recognized landmarks may be stored on aremote server, and processing unit 110 may instruct a wireless system ofvehicle 200, which may include a wireless transceiver, to retrieve theinformation associated with the recognized landmarks. In other cases,the information may already reside on vehicle 200 (e.g., within a localmap from sparse data map 800 received during navigation or within asparse data map 800 preloaded into memory of vehicle 200). In someembodiments, this positional information may be used to calibrate one ormore indicators of speed of an autonomous vehicle (e.g., one or morespeed sensors of vehicle 200).

FIG. 70 is a flowchart showing an exemplary process 7000 for calibratinga speed of vehicle 200 consistent with disclosed embodiments. At step7002, processing unit 110 may receive from an image capture device 122 aplurality of images representative of an environment of vehicle 200. Insome embodiments, images may be captured at different times by imagecapture device 122 (e.g., images may be captured many times per second,for example). In some embodiments, vehicle 200 may include a pluralityof image capture devices (e.g., image capture devices 122 and 124 ofvehicle 200), and processing unit 110 may receive from each imagecapture device, a plurality of images representative of an environmentof vehicle 200. The plurality of images received from each image capturedevice may include images captured at different times by one or more ofthe image capture devices on the vehicle.

At step 7004, processing unit 110 may analyze the plurality of images toidentify at least two recognized landmarks present in the images. Thetwo recognized landmarks need not be present in a single image fromamong the plurality of images. In fact, in many cases, the tworecognized landmarks identified in the plurality of images will notappear in the same images. Rather, a first recognized landmark may beidentified in a first image received from an image capture device. At alater time, and perhaps many image frames later (e.g., 10 s, 100 s, or1000 s of image frames later, or more), a second recognized landmark maybe identified in another of the plurality of images received from theimage capture device. The first recognized landmark may be used todetermine a first location S1 of the vehicle along a target trajectoryat time T1, and the second recognized landmark may be used to determinea second location S2 of the vehicle along the target trajectory at timeT2. Using information such as a measured distance between S1 and S2 andknowing a time difference between T1 and T2 may enable the processorunit of the vehicle to determine a speed over which the distance betweenS1 and S2 was covered. This speed can be compared to an integratedvelocity obtained based on an output of the vehicle's speed sensor. Insome embodiments, this comparison may yield a correction factor neededto adjust/calibrate the vehicle's speed sensor to match the speeddetermined based on the S1 to S2 speed calculation.

Alternatively, or additionally, the processor unit may use an output ofthe vehicle's speed sensor to determine a sensor-based distance readingbetween S1 and S2. This sensor based distance reading can be compared toa calculated distance between S1 and S2 in order to determine anappropriate correction factor to calibrate the vehicle's speed sensor.

Processing unit 110 may identify recognized landmarks in a capturedimage stream according to any of the techniques described elsewhere inthe disclosure. For example, processing unit 110 may compare one or moreobserved characteristics of a potential landmark to characteristics fora recognized landmark stored in sparse data map 800. Where one or moreof the observed characteristics is found to match the storedcharacteristics, then processing unit 110 may conclude that the observedpotential landmark is, in fact, a recognized landmark. Suchcharacteristics may include, among other things, size, shape, location,distance to another recognized landmark, landmark type, condensed imagesignature, etc.

At step 7006, processing unit 110 may determine, based on knownlocations of the two recognized landmarks, a value indicative of adistance between the at least two recognized landmarks. For example, asdiscussed above, processing unit 110 may retrieve or otherwise rely uponinformation associated with the recognized landmarks after identifyingthe recognized landmarks. Further, the information may includepositional information of the recognized landmarks, and processing unit110 may compute a distance between the two recognized landmarks based onthe retrieved positional information associated with the two landmarks.Positional information may include, for example, global coordinates ofeach recognized landmark determined, for example, based on anaggregation of position determinations (e.g., GPS based positiondeterminations) made by a plurality of vehicles upon prior traversalsalong the road segments including the two recognized landmarks.

At step 7008, processing unit 110 may determine, based on an output ofat least one sensor associated with the autonomous vehicle, a measureddistance between the at least two landmarks. In some embodiments,processing unit 110 may use an odometry technique based on imagescaptured by image capture device 122, inertial sensors, and/or aspeedometer of vehicle 200 to measure the distance between the tworecognized landmarks. For example, as noted above, a first position ofthe vehicle S1 may be used as a starting point and a second position ofthe vehicle S2 may be used as an ending point. These positions may bedetermined based on images collected of the first and second recognizedlandmarks, respectively, using techniques described in other sections ofthe disclosure. The vehicle sensors (e.g., the speedometer) can be usedto measure a distance between location S1 and S2. This measured distancemay be compared to a calculated distance between locations S1 and S2,for example, along a predetermined target trajectory of the vehicle.

In some embodiments, S1 and S2 may be selected according to a particularrelationship with the recognized landmarks. For example, S1 and S2 maybe selected as locations where lines extending from the first and secondlandmarks, respectively, intersect the target trajectory at rightangles. Of course, any other suitable relationship may also be used. Insuch embodiments, where S2 and S1 are defined according to apredetermined relationship, a distance between S2 and S1 may be knownand represented, for example, in sparse data map 800 (e.g., as adistance value to the preceding recognized landmark). Thus, rather thanhaving to calculate a distance between S1 and S2, in such embodiments,this distance value may already be available from sparse data map 800.As in previous embodiments, the predetermined distance between S1 and S2may be compared to the distance between S1 and S2 measured using thevehicle sensors.

For example, in some embodiments, measuring the distance between the twolandmarks may be done via a GPS device (e.g., position sensor 130). Forexample, two landmarks may be selected, which are distant from eachother (e.g., 5 km) and the road between them may be rather straight. Alength of that road segment may be measured, for example, by subtractingthe GPS coordinates of the two landmarks. Each such coordinate may bemeasured with an error of a few meters (i.e., the GPS error), but due tothe long length of the road segment this may be a relatively smallerror.

At step 7010, processing unit 110 may determine a correction factor forthe at least one sensor based on a comparison of the value indicative ofthe distance between the at least two recognized landmarks and themeasured distance between the at least two landmarks. The correctionalfactor may be, for example, a ratio of the value indicative of thedistance between the at least to recognized landmarks and the measureddistance between the at least two landmarks. In some embodiments, thecorrection factor may be referred to as a calibration factor and mayrepresent a value that may be used to transform the measured distancevalue based on the vehicle's sensors into the calculated/predetermineddistance value.

In an optional step, processing unit 110 may determine a compositecorrection factor based on a plurality of determined correction factors.Correction factors of the plurality of determined correction factors maybe determined based on different set of landmarks. In some embodiments,the composite correction factor is determined by averaging the pluralityof determined correction factors or by finding a mean of the pluralityof determined correction factors.

FIG. 71 is a diagrammatic top view representation of exemplary vehicle200 including a system for calibrating an indicator of speed of thevehicle consistent with disclosed embodiments. In the example of FIG.71, vehicle 200 is traveling on a first road segment 7102A. FIG. 71 alsoshows a second road segment 7102B and lane marks 7104, 7106. A roadsegment is includes any portion of a road.

In some embodiments, processing unit 110 may determine a distance alonga road segment (e.g., road segments 7102A or 7102B) using one or moresensors of vehicle 200. In one example, processing unit 110 maydetermine, using one or more sensors of vehicle 200, a road signatureprofile associated with the road segment vehicle 200 is traveling on(e.g., road segment 7102A). Such road signature profile may beassociated with any discernible/measurable variation in at least oneparameter associated with the road segment. In some cases, such profilemay be associated with, for example, variations in surface roughness ofa particular road segment, variations in road width over a particularroad segment, variations in distances between dashed lines painted alonga particular road segment, variations in road curvature along aparticular road segment, etc. As discussed above, FIG. 11D showsexemplary road signature profile 1160. While a road signature profilemay represent any of the parameters mentioned above, or others, in oneexample, the road signature profile may represent a measure of roadsurface roughness, as obtained, for example, by monitoring one or moresensors providing outputs indicative of an amount of suspensiondisplacement as vehicle 200 travels on first road segment 7102A.Alternatively, road signature profile 1160 may represent variation inroad width, as determined based on image data obtained via image capturedevice 122 of vehicle 200 traveling on first road segment 7102A. Suchprofile may be useful, for example, in determining a particular locationof an autonomous vehicle relative to a particular target trajectory.That is, as it traverses a road segment, an autonomous vehicle maymeasure a profile associated with one or more parameters associated withthe road segment. If the measured profile can be correlated/matched witha predetermined profile that plots the parameter variation with respectto position along the road segment, then the measured and predeterminedprofiles may be used (e.g., by overlaying corresponding sections of themeasured and predetermined profiles) in order to determine a currentposition along the road segment and, therefore, a current positionrelative to a target trajectory for the road segment. A distance along aroad segment may be determined based on a plurality of positionsdetermined along a road segment.

FIG. 72 is a flowchart showing exemplary process 7200 for calibrating anindicator of speed of vehicle 200 consistent with disclosed embodiments.In some embodiments, vehicle 200 may calibrate the indicator of speed ofvehicle 200 by calculating a correction factor based on a distancedetermined along the road segment and a distance value received via thewireless transceiver. That is, rather than determining positions S1 andS2 based on landmarks and then calculating a distance between positionsS1 and S2, a distance value for a predetermined portion of a roadsegment may be received via sparse data map 800 (e.g., via a wirelesstransceiver).

At step 7204, processing unit 110 may receive, via a wirelesstransceiver, a distance value associated with the road segment. In oneexample, the wireless transceiver may be a 3GPP-compatible or anLTE-compatible transceiver. The distance value associated with the roadsegment stored on the remote server may be determined based on priormeasurements made by a plurality of measuring vehicles. For example, aplurality of vehicles may have previously traveled on the same roadsegment in the past and uploaded the determined distance valuesassociated with the road segment (e.g., between two or morepredetermined reference points, landmarks, etc.) to the remote server.The distance value associated with the road segment stored on the remoteserver may be an average of the distance values determined by theplurality of measuring vehicles.

In some embodiments, the distance value associated with the road segmentstored on the remote server may be determined based on priormeasurements made by at least 100 measuring vehicles. In otherembodiments, the distance value associated with the road segment storedon the remote server may be determined based on prior measurements madeby at least 1000 measuring vehicles.

At step 7206, processing unit 110 may determine a correction factor forthe at least one speed sensor based on the determined distance along theroad segment and the distance value received via the wirelesstransceiver. The correctional factor may be, for example, a ratio of thedistance along the road segment determined using a sensor and thedistance value received via the wireless transceiver. And, thecorrection factor may represent a value that may be used to transformthe measured distance value based on the vehicle's sensors into thereceived/predetermined distance value.

In an optional step, processing unit 110 may determine a compositecorrection factor based on a plurality of determined correction factors.Correction factors of the plurality of determined correction factors maybe determined based on different landmarks. In some embodiments, thecomposite correction factor is determined by averaging the plurality ofdetermined correction factors or by finding a mean of the plurality ofdetermined correction factors.

Determining Lane Assignment Based on Recognized Landmark Location

In addition to determining a lane assignment based on analysis of acamera output (e.g., seeing additional lanes to the right and/or left ofa current lane of travel for the vehicle), the system may determineand/or validate a lane assignment based on a determined lateral positionof recognized landmarks relative to the vehicle.

FIG. 73 is a diagrammatic illustration of a street view of an exemplaryroad segment, consistent with disclosed embodiments. As shown in FIG.73, road segment 7300 may include a number of components, including road7310, lane marker 7320, landmarks 7330, 7340, and 7350, etc. In additionto the components depicted in exemplary road segment 7300, a roadsegment may include other components, including fewer or additionallanes, landmarks, etc., as would be understood by one of ordinary skillin the art.

In one embodiment, road segment 7300 may include road 7310, which may bedivided by one or more lane markers 7320 into two or more lanes. Roadsegment 7300 may also include one or more vehicles, such as vehicle7360. Moreover, road segment 7300 may include one or more landmarks,such as landmarks 7330, 7340, and 7350. In one embodiment, such as shownin FIG. 73, landmarks may be placed alongside road 7310. Landmarksplaced alongside road 7310 may include, for example, traffic signs(e.g., speed limit signs, such as landmarks 7330 and 7340), mile markers(e.g., landmark 7350), billboards, exit signs, etc. Landmarks may alsoinclude general purpose signs (e.g., non-semantic signs relating tobusinesses or information sources, etc.). Alternatively, landmarks maybe placed on or above road 7310. Landmarks placed on or above road 7310may include, for example, lane markers (e.g., lane marker 7320),reflectors, exit signs, marquees, etc. Landmarks can also include any ofthe examples discussed elsewhere in this disclosure.

FIG. 74 is a diagrammatic illustration of a birds-eye view of anexemplary road segment, consistent with disclosed embodiments. As shownin FIG. 74, exemplary road segment 7400 may include a number ofcomponents, including road 7405, lane marker 7410, vehicle 7415,traversed path 7420, heading 7425, predicted path 7430, predeterminedroad model trajectory 7435, landmarks 7440, 7455, and 7470, directoffset distances 7445, 7460, and 7475, and lateral offset distances 7450and 7465. In addition to the components depicted in exemplary roadsegment 7300, a road segment may include other components, includingfewer or additional lanes, landmarks, and vehicles, as would beunderstood by one of ordinary skill in the art.

In one embodiment, road segment 7400 may include road 7405, which may bedivided by one or more lane markers 7410 into two or more lanes. Roadsegment 7300 may also include one or more vehicles, such as vehicle7415. Moreover, road segment 7400 may include one or more landmarks,such as landmarks 7440, 7455, and 7470.

In one embodiment, vehicle 7415 may travel along one or more lanes ofroad 7405 in a path. The path that vehicle 7415 has already traveled isrepresented in FIG. 74 as traversed path 7420. The direction in whichvehicle 7415 is headed is depicted as heading 7425. Based on the currentlocation of vehicle 7145 and heading 7425, among other factors, a paththat vehicle 7415 is expected to travel, such as predicted path 7430,may be determined. FIG. 74 also depicts predetermined road modeltrajectory 7435, which may represent an ideal path for vehicle 7415.

In one embodiment, direct offset distances 7445, 7460, and 7475 mayrepresent the distance between vehicle 7415 and landmarks 7440, 7455,and 7470, respectively. Lateral offset distances 7450 and 7465 mayrepresent the distance between vehicle 7415 and the landmarks 7440 and7455 when vehicle 7415 is directly alongside those landmarks.

For example, two techniques may be used to calculate the number of lanesbased on the lateral distance estimation between the host vehicle andthe landmark. As a first example, a clustering technique may be used.Using mean-shift clustering, the system may calculate the number oflanes and the lane assignment for each drive. Next, for enriching thenumber of observations and to provide observations from each lane, thesystem may add observations for the adjacent lanes (e.g., if the lanes'DNN networks decided there are such lanes). Next, the system maydetermine the road width and splitting it into lanes based on thecalculated lane width. As a second example, in another technique, basedon sightings of vehicles where the lanes' DNN network determined theyare either on the extreme (left or right) lane or on the one adjacent toit, the system may create a set of estimations of the lateral distancebetween the land mark and the extreme left or right lane mark. Next,using either a voting or a least squares mechanism, the system maydetermine an agreed distance estimation between the land mark and theroad edges. Next, from the distance estimates to the road edges, thesystem may extract the road width, and determine the number of lanes bydividing the road width by the median lane width observed in the drives.The system may assign a lane to each drive based on which bin theobserved distance between the host.

FIG. 75 is a flowchart showing an exemplary process 7500 for determininga lane assignment for a vehicle (which may be an autonomous vehicle)along a road segment, consistent with disclosed embodiments. The stepsassociated with this exemplary process may be performed by thecomponents of FIG. 1. For example, the steps associated with the processmay be performed by application processor 180 and/or image processor 190of system 100 illustrated in FIG. 1.

In step 7510, at least one processor receives from a camera at least oneimage representative of an environment of the vehicle. For example,image processor 128 may receive one or more images from one or more ofcameras 122, 124, and 126 representing an environment of the vehicle.Image processor 128 may provide the one or more images to applicationprocessor 180 for further analysis. The environment of the vehicle mayinclude the area surrounding the exterior of the vehicle, such as theroad segment and any signs, buildings, or landscaping along the roadsegment. In one embodiment, the environment of the vehicle includes theroad segment, a number of lanes, and the at least one recognizedlandmark.

In step 7520, the at least one processor analyzes the at least one imageto identify at least one recognized landmark. In one embodiment, the atleast one recognized landmark includes at least one of a traffic sign,an arrow marking, a lane marking, a dashed lane marking, a trafficlight, a stop line, a directional sign, a reflector, a landmark beacon,or a lamppost, etc. For example, the at least one recognized landmarkmay include landmarks 7330, 7340, and 7350, each of which is a trafficsign. In particular, landmarks 7330 and 7340 are speed limit signs, andlandmark 7350 is a mile marker sign. In another embodiment, the at leastone recognized landmark includes a sign for a business. For example, theat least one recognized landmark may include a billboard advertisementfor a business or a sign marking the location of a business.

In step 7530, the at least one processor determines an indicator of alateral offset distance between the vehicle and the at least onerecognized landmark. In some embodiments, the determination of theindicator of the lateral offset distance between the vehicle and the atleast one recognized landmark may be based on a known position of the atleast one recognized landmark. The known position of the at least onerecognized landmark may be stored, for example, in memory 140 or mapdatabase 160 (e.g., as part of sparse map 800).

In step 7540, the at least one processor determines a lane assignment ofthe vehicle along the road segment based on the indicator of the lateraloffset distance between the vehicle and the at least one recognizedlandmark. For example, the at least one processor may determine whichlane the vehicle is travelling in based the indicator of lateral offsetdistance. For example, a lane assignment may be determined based onknowledge of a lateral distance from the recognized landmark to a laneedge closest to the recognized landmark, to any lane edges present onthe road, to a target trajectory associated with a road segment, or tomultiple target trajectories associate with the road segment, etc. Thedetermined indicator of lateral offset distance between the recognizedlandmark and the host vehicle may be compared to any of thesequantities, among others, and then used to determine a current laneassignment based on one or more arithmetic and/or trigonometriccalculations.

In one embodiment, the at least one recognized landmark includes a firstrecognized landmark on a first side of the vehicle and a secondrecognized landmark on a second side of the vehicle and whereindetermination of the lane assignment of the vehicle along the roadsegment is based on a first indicator of lateral offset distance betweenthe vehicle and the first recognized landmark and a second indicator oflateral offset distance between the vehicle and the second recognizedlandmark. The lane assignment may be determined based on a ratio of thefirst indicator of lateral offset distance to the second indicator oflateral offset distance. For example, if the vehicle is located 20 feetfrom a landmark posted on the left edge of the road and 60 feet from alandmark posted on the right edge of the road, then the lane assignmentmay be determined based on this ratio, given information on the numberof lanes on the road segment or lane width. Alternatively, the laneassignment may be calculated separately based on the indicators oflateral offset distance between the vehicle and the first and secondrecognized landmarks, and these separate calculations may be checkedagainst one another to verify that the determined lane assignment(s) arecorrect.

Super Landmarks as Navigation Aids

The system may navigate by using recognized landmarks to aid indetermining a current location of an autonomous vehicle along a roadmodel trajectory. In some situations, however, landmark identity may beambiguous (e.g., where there is a high density of similar types oflandmarks). In such situations, landmarks may be grouped together to aidin their recognition. For example, distances between landmarks within agroup of landmarks may be used to create a super landmark signature toaid in positive identification of the landmarks. Other characteristics,such as landmark sequences within a group of landmarks, may also beused.

FIG. 76 is an illustration of a street view of an exemplary roadsegment, consistent with disclosed embodiments. As shown in FIG. 76,road segment 7600 may include a number of components, including road7610, lane marker 7620, vehicle 7630, and landmarks 7640, 7650, 7660,7670, and 7680. In addition to the components depicted in exemplary roadsegment 7600, a road segment may include other components, includingfewer or additional lanes, landmarks, and vehicles, as would beunderstood by one of ordinary skill in the art.

In one embodiment, road segment 7600 may include road 7610, which may bedivided by one or more lane markers 7620 into two or more lanes. Roadsegment 7600 may also include one or more vehicles, such as vehicle7630. Moreover, road segment 7600 may include one or more landmarks,such as landmarks 7640, 7650, 7660, 7670, and 7680. In one embodiment,landmarks may be assigned to structures/objects associated with road7610 (e.g., landmarks 7670 and 7680). Landmarks along road 7610 mayinclude, for example, traffic signs (e.g., mile markers, such aslandmark 7670), billboards (e.g., landmark 7680), lane markers (e.g.,landmark 7620), reflectors, traffic signs (e.g., exit signs, such aslandmarks 7640, 7650, and 7660), marquees, etc. Landmarks identified orotherwise represented in sparse data map 800 may be referred to asrecognized landmarks.

Some areas, especially in urban environments, may have high densities ofrecognized landmarks. Thus, in some cases, distinguishing betweencertain recognized landmarks may be difficult based on comparisons basedsolely on landmark size, shape, type, indexed location, etc. To furtheraid in identifying one or more recognized landmarks from within imagescaptured of a vehicle's environment, a group of two or more landmarksmay be designated as a super landmark. Such a super landmark may offeradditional characteristics that may aid in identifying or verifying oneor more recognized landmarks (e.g., from among the group of landmarks).

In FIG. 76, for example, a super landmark may be formed from the groupconsisting of landmarks 7640, 7650, 7660, 7670, and 7680, or some subsetof two or more of those landmarks. By grouping two or more landmarkstogether, the probability of accurately identifying constituentlandmarks from a distant vantage point may be increased.

A super landmark may be associated with one or more characteristics,such as distances between constituent landmarks, a number of landmarksin the group, an ordering sequence, one or more relative spatialrelationships between the members of the landmark group, etc. Moreover,these characteristics may be used to generate a super landmarksignature. The super landmark signature may represent a unique faun ofidentifying the group of landmarks or even a single landmark within thegroup.

FIG. 77A is an illustration of birds-eye view of an exemplary roadsegment, consistent with disclosed embodiments. As shown in FIG. 77,exemplary road segment 7700 may be associated with a number ofcomponents, including road 7705, lane marker 7710, vehicle 7715,traversed path 7720, predetermined road model trajectory 7725, landmarks7730, 7735, 7740, 7745, and 7750, lateral offset vector 7755, and directoffset vector 7760. In addition to the components depicted in exemplaryroad segment 7700, a road segment may be associated with othercomponents, including fewer or additional lanes, landmarks, andvehicles, as would be understood by one of ordinary skill in the art.

In one embodiment, road segment 7700 may include road 7705, which may bedivided by one or more lane markers 7710 into two or more lanes. Roadsegment 7700 may also include one or more vehicles, such as vehicle7715. Moreover, road segment 7700 may include one or more landmarks,such as landmarks 7730, 7735, 7740, 7745, and 7750.

In one embodiment, vehicle 7715 may travel along one or more lanes ofroad 7705 in a path. The path that vehicle 7715 has already traveled isrepresented in FIG. 77 as traversed path 7720. FIG. 77 also depictspredetermined road model trajectory 7725, which may represent a targetpath for vehicle 7715.

In one embodiment, a direct offset vector may be a vector connectingvehicle 7715 and a landmark. For example, direct offset vector 7760 maybe a vector connecting vehicle 7715 and landmark 7730. The distancebetween vehicle 7715 and a landmark may be equivalent to the magnitudeof direct offset vector connecting vehicle 7715 with the landmark. Alateral offset vector may be a vector connecting vehicle 7715 with apoint on the side of the road in line with a landmark. The lateraloffset distance for a vehicle with respect to a landmark may beequivalent to the magnitude of the lateral offset vector and, further,may be equivalent to the distance between vehicle 7715 and the landmarkwhen vehicle 7715 is directly alongside the landmark. The lateral offsetdistance between vehicle 7715 and a landmark may be computed bydetermining a sum of a first distance between the vehicle and the edgeof the road on which the landmark is located and a second distancebetween that edge and the landmark.

FIG. 77B provides a street level view of a road segment including asuper landmark made up of four recognized landmarks: a speed limit sign7790, a stop sign 7791, and two traffic lights 7792 and 7793. Any of therecognized landmarks included in the super landmark group may beidentified based on recognition of various relationships between thelandmarks included in the group. For example, a sequence, which may bestored in sparse data map 800, of a speed limit sign at a distance D1,followed by a stop sign at a distance D2, and two traffic lights at adistance D3 from a host vehicle (where D3>D2>D1) may constitute aunique, recognizable characteristic of the super landmark that may aidin verifying speed limit sign 7790, for example, as a recognizedlandmark from sparse data map 800.

Other relationships between the members of a super landmark may also bestored in sparse data map 800. For example, at a particularpredetermined distance from recognized landmark 7790 and along a targettrajectory associated with the road segment, the super landmark may forma polynomial 7794 between points A, B, C, and D each associated with acenter of a member of the super landmark. The segment lengths A-B, B-C,C-D, and D-A may be determined and stored in sparse data map 800 for oneor more positions relative to the location of the super landmark.Additionally, a triangle 7795 may be formed by traffic light 7793,traffic light 7792, and stop sign 7791. Again, the lengths of the sidesas well as angles of triangle 7795 may be referenced in sparse data map800 for one or more positions relative to the location of the superlandmark. Similar information may be determined and stored for atriangles 7796 (between points A, C, and D) and 7797 (between pointsA-B-C). Such angles, shapes, and segment lengths may aid in recognitionof a super landmark from a certain viewing location relative to thesuper landmark. For example, once the vehicle is located at a viewinglocation for which visual information for the super landmark is includedin sparse data map 800, the processing unit of the vehicle can analyzeimages captured by one or more cameras onboard the vehicle to look forexpected shapes, patterns, angles, segment lengths, etc. to determinewhether a group of objects forms an expected super landmark. Uponverifying the recognized super landmark, position determinations for thevehicle along a target trajectory may commence based on any of thelandmarks included in a super landmark group.

FIG. 78 is a flowchart showing an exemplary process 7800 forautonomously navigating a vehicle along a road segment, consistent withdisclosed embodiments. The steps associated with this exemplary processmay be performed by the components of FIG. 1. For example, the stepsassociated with the process may be performed by application processor180 and/or image processor 190 of system 100 illustrated in FIG. 1.

In step 7810, at least one processor may receive from a camera at leastone image representative of an environment of the vehicle. For example,image processor 128 may receive one or more images from one or more ofcameras 122, 124, and 126 representing an environment of the vehicle.Image processor 128 may provide the one or more images to applicationprocessor 180 for further analysis. The environment of the vehicle mayinclude the area surrounding the exterior of the vehicle, such as theroad segment and any signs, buildings, or landscaping along the roadsegment. In one embodiment, the environment of the vehicle includes theroad segment, a number of lanes, and the at least one recognizedlandmark.

In step 7820, the at least one processor may analyze the at least oneimage to identify a super landmark and identify at least one recognizedlandmark from the super landmark. In one embodiment, the at least onerecognized landmark includes at least one of a traffic sign, an arrowmarking, a lane marking, a dashed lane marking, a traffic light, a stopline, a directional sign, a reflector, a landmark beacon, or a lamppost.For example, the at least one recognized landmark may include landmarks7640, 7650, 7660, and 7670, each of which is a traffic sign. Inparticular, landmarks 7640, 7650, and 7660 are exit signs, and landmark7670 is a mile marker sign. In another embodiment, the at least onerecognized landmark includes a sign for a business. For example, the atleast one recognized landmark may include a billboard advertisement fora business (e.g., landmark 7680) or a sign marking the location of abusiness.

As noted above, identification of the at least one landmark is based, atleast in part, upon one or more landmark group characteristicsassociated with the group of landmarks. In one embodiment, the one ormore landmark group characteristics may include relative distancesbetween members of the group of landmarks. For example, the landmarkgroup characteristics may include information that specifies thedistance that separates each landmark in the group from each of theother landmarks in the group. In another embodiment, the one or morelandmark group characteristics may include an ordering sequence ofmembers of the group of landmarks. For example, the group of landmarksmay be associated with a sequence indicating the order in which thelandmarks appear from left to right, front to back, etc., when viewedfrom the road. In yet another embodiment, the one or more landmark groupcharacteristics may include a number of landmarks included in the groupof landmarks.

Referring to FIG. 76 as an example, a landmark group (or super landmark)may consist of landmarks 7640, 7650, 7660, 7670, and 7680. This landmarkgroup may be associated with landmark group characteristics, includingthe relative distances between each landmark and each of the otherlandmarks in the group, an ordering sequence of landmarks in the group,and a number of landmarks. In the example depicted in FIG. 76, thelandmark group characteristics may include information that specifiesthe distance between landmark 7680 and each of landmarks 7640, 7650,7660, and 7670, the distance between landmark 7640 and each of landmarks7650, 7660, and 7670, the distance between landmark 7650 and each oflandmarks 7660 and 7670, and the distance between landmarks 7660 and7670.

Further, in this example, an ordering sequence may indicate that theorder of landmarks in the group from left to right (when viewed from theperspective of a vehicle driving along the road, e.g., vehicle 7630) is7680, 7640, 7650, 7660, and 7670. Alternatively or additionally, theordering sequence may indicate that the order of landmarks in the groupfrom front to back (e.g., earliest to latest traversed in a path alongthe road) is first 7670, then 7640, 7650, and 7660, and last 7680.Moreover, the landmark group characteristics may specify that thisexemplary landmark group includes five landmarks.

In one embodiment, identification of the at least one landmark may bebased, at least in part, upon a super landmark signature associated withthe group of landmarks. A super landmark signature may be a signaturefor uniquely identifying a group of landmarks. In one embodiment, asuper landmark signature may be based on one or more of the landmarkgroup characteristics discussed above (e.g., number of landmarks,relative distance between landmarks, and ordering sequence oflandmarks).

Once a recognized landmark is identified based on an identifiedcharacteristic of the super landmark group, predeterminedcharacteristics of the recognized landmark may be used to assist a hostvehicle in navigation. For example, in some embodiments, the recognizedlandmark may be used to determine a current position of the hostvehicle. In some cases, the current position of the host vehicle may bedetermined relative to a target trajectory from sparse data model 800.Knowing the current position relative to a target trajectory may aid indetermining a steering angle needed to cause the vehicle to follow thetarget trajectory (for example, by comparing a heading direction to adirection of the target trajectory at the determined current position ofthe vehicle relative to the target trajectory).

A position of the vehicle relative to a target trajectory from sparsedata map 800 may be determined in a variety of ways. For example, insome embodiments, a 6D Kalman filtering technique may be employed. Inother embodiments, a directional indicator may be used relative to thevehicle and the recognized landmark. For example, in step 7830, the atleast one processor may determine, relative to the vehicle, adirectional indicator associated with the at least one landmark. In oneembodiment, the directional indicator may include a line or vectorconnecting the vehicle and the at least one landmark. The directionalindicator may indicate the direction in which the vehicle would have totravel to arrive at the at least one landmark. For example, in theexemplary embodiment depicted in FIG. 77, direct offset vector 7760 mayrepresent a directional indicator associated with landmark 7730 relativeto vehicle 7715.

In step 7840, the at least one processor may determine an intersectionof the directional indicator with a predetermined road model trajectoryassociated with the road segment. In one embodiment, the predeterminedroad model trajectory may include a three-dimensional polynomialrepresentation of a target trajectory along the road segment. The targettrajectory may include an ideal trajectory for the vehicle for aspecific location along the road segment. In one embodiment, the atleast one processor may further be programmed to determine a locationalong the predetermined road model trajectory based on a vehiclevelocity. For example, the at least one processor may access informationthe location and velocity of the vehicle at a specific time, compute anestimated distance traveled based on the velocity and time passed sincethe vehicle was at that location, and identify a point along thepredetermined road model trajectory that is the estimated distancebeyond the previously observed location.

In step 7850, the at least one processor may determine an autonomoussteering action for the vehicle based on a direction of thepredetermined road model trajectory at the determined intersection. Inone embodiment, determining an autonomous steering action for thevehicle may include comparing a heading direction of the vehicle to thepredetermined road model trajectory at the determined intersection. Inone embodiment, the autonomous steering action for the vehicle mayinclude changing the heading of the vehicle. In another embodiment, theautonomous steering action for the vehicle may include changing thespeed of the vehicle by applying the gas or brake to accelerate ordecelerate, respectively.

Adaptive Autonomous Navigation

In some embodiments, the disclosed systems and methods may provideadaptive autonomous navigation and update a sparse map. For example, thedisclosed systems and methods may adapt navigation based on userintervention, provide adapt navigate based on determinations made by thesystem (e.g., a self-aware system), adapt a road model based on whetherobserved conditions on a road are transient or non-transient (e.g., anadaptive road model manager), and manage a road model based on selectivefeedback received from one or more systems. These adaptive systems andmethods are discussed in further detail below.

Adaptive Navigation Based on User Intervention

In some embodiments, the disclosed systems and methods may involveadaptive navigation based on user intervention. For example, asdiscussed in earlier sections, a road model assembled based upon inputfrom existing vehicles may be distributed from a server (e.g., server1230, discussed earlier) to vehicles. Based on feedback received fromautonomous vehicles, the system may determine whether one or moreupdates (e.g., adaptations to the model) are needed to the road model toaccount for changes in road situations, for example. For example, insome embodiments, a user may intervene to alter a maneuver of a vehicle(which may be an autonomous vehicle) while the vehicle is traveling on aroadway according to the road model. An altered maneuver of the vehiclebased on user intervention may be made in contradistinction to overridepredetermined vehicular trajectory instructions provided by the roadmodel. Further, the disclosed systems and methods may capture and storenavigational situation information about the situation in which theoverride occurred and/or send the navigational situation informationfrom the vehicle to the server over one or more networks (e.g., over acellular network and/or the Internet, etc.) for analysis. As discussedherein, navigational situation information may include one or more of alocation of a vehicle, a distance of a vehicle to a recognized landmark,an observed condition, a time of day, an image or a video captured by animage capture device of a vehicle, or any other suitable informationalsource regarding a navigational situation.

FIG. 79A illustrates a plan view of vehicle 7902 traveling on a roadway7900 approaching wintery and icy road conditions 7930 at a particularlocation consistent with disclosed embodiments. Vehicle 7902 may includea system that provides navigation features, including features thatadapt navigation based on user intervention. Vehicle 7902 may includecomponents such as those discussed above in connection with vehicle 200.For example, as depicted, vehicle 7902 may be equipped with imagecapture devices 122 and 124; more or fewer image capture devices(including cameras, for example) may be employed.

As shown, roadway 7900 may be subdivided into lanes, such as lanes 7910and 7920. Lanes 7910 and 7920 are shown as examples; a given roadway7900 may have additional lanes based on the size and nature of theroadway, for example, an interstate highway. In the example of FIG. 79A,vehicle 7902 is traveling in lane 7910 according to instructions derivedfrom the road model (e.g., a heading direction along a targettrajectory) and approaching wintery and icy road conditions 7930 at aparticular vehicle location as identified by, e.g., position sensor 130,a temperature sensor, and/or an ice sensor. Where a user intervenes inorder to override autonomously generated steering instructions (e.g.,those enabling the vehicle to maintain a course along the targettrajectory) and alter the course of the vehicle 7902 traveling in lane7910 (e.g., to turn due to the icy conditions), processing unit 110 maystore navigational situation information and/or send the navigationalsituation information to a server of the road model system for use inmaking a possible update. In this example, the navigational situationinformation may include a location of the vehicle identified by positionsensor 130 or based on a landmark-based determination of position alonga target trajectory, an image captured by an image capture deviceincluded in the vehicle depicting the vehicle's environment, an imagestream (e.g., a video), sensor output data (e.g., from speedometers,accelerometers, etc.).

In some embodiments, processing unit 110 may send the navigationalsituational information from the vehicle to the server via a wirelessdata connection over one or more networks (e.g., over a cellular networkand/or the Internet, etc.). The server side may analyze the receivedinformation (e.g., using automated image analysis processes) todetermine whether any updates to sparse data model 800 are warrantedbased on the detected user intervention. In this example, the server mayrecognize the presence of wintery or icy road conditions in the images(a temporary or transient condition) and, therefore, may determine notto change or update the road model.

FIG. 79B illustrates a plan view of vehicle 7902 traveling on a roadwayapproaching a pedestrian consistent with disclosed embodiments. In theexample of FIG. 79B, vehicle 7902 is driving in lane 7910 of roadway7900 with a pedestrian 7922. As shown, pedestrian 7922 may suddenlybecome positioned directly in the roadway 7900 crossing either lane 7910or 7920. In this example, when a user intervenes to override the roadmodel in order to avoid the pedestrian and alter the maneuver of thevehicle 7902 traveling in lane 7910 along a target trajectory associatedwith the road segment, navigational situation information including aposition of the vehicle along a target trajectory for a road segment(e.g., determined based on a distance d₁ to a recognized landmark, suchas speed limit sign 7923), video or images including capturingconditions of the vehicle's surroundings during the user intervention,sensor data, etc. In example shown in FIG. 49B, given the temporarynature of a crossing pedestrian the server may determine not change orupdate the road model.

Although the example shown in FIG. 79B depicts speed limit sign 7923,other recognized landmarks (not shown) may be used. Landmarks mayinclude, for example, any identifiable, fixed object in an environmentof at least one road segment or any observable characteristic associatedwith a particular section of the road segment. In some cases, landmarksmay include traffic signs (e.g., speed limit signs, hazard signs, etc.).In other cases, landmarks may include road characteristic profilesassociated with a particular section of a road segment. Further examplesof various types of landmarks are discussed in previous sections, andsome landmark examples are shown in FIG. 10.

FIG. 79C illustrates a plan view of a vehicle traveling on a roadway inclose proximity to another vehicle consistent with disclosedembodiments. In the example of FIG. 79C, two vehicles 7902 a and 7902 bare driving in lane 7910 of roadway 7900. As shown, vehicle 7902 b hassuddenly driven directly in front of vehicle 7902 a in lane 7910 ofroadway 7900. Where a user intervenes to override the road model andalter the course of the vehicle 7902 a traveling in lane 7910 (e.g., toturn due to the proximate vehicle), navigational situation informationmay be captured and stored in memory (e.g., memory 140) and/or sent to aserver (e.g., server 1230) for making a possible update to the roadmodel. For example, in this example, the navigational situationinformation may include a location of vehicle 7902 a. The navigationalsituation information may further include one more images depicting theenvironment of vehicle 7902 at the time of the user intervention. Giventhe temporary nature of another contiguous or proximate vehicle,however, the server may not change or update the road model.

FIG. 79D illustrates a plan view of a vehicle traveling on a roadway ina lane that is ending consistent with disclosed embodiments. Vehicle7902 may receive from image capture devices 122 and 124 at least oneenvironmental image of a turning roadway 7900 representative of a lane7910 ending. Lane 7910 may be ending based on a recent change to lane7910 resulting an abrupt shortening distance of d₂. For example, thelane may be ending as a result of recently positioned concrete barriersat the site of a construction zone. As a result of this unexpectedshortening, a user may intervene to change the course of vehicle 7902 inview of the change to lane 7910. As will be discussed in more detail inanother section, it is also possible for processing unit 110 torecognize the ending lane (e.g., based on captured images of concretebarriers in front of the vehicle) and automatically adjust the course ofthe vehicle and send navigational situation information to the serverfor use in possible updates to sparse data model 800. As a result of theuser intervention, the system may measure distances (such as c₁ and c₂).For example, distances c₁ and c₂ may represent the distance from a sideof vehicle 7902 to the edge of lane 7910, be it lane constraint 7924 orthe dashed center line in the middle of roadway 7900 dividing lanes7910/7920. In other embodiments, a distance may be measured to laneconstraint 7924 on the far side of lane 7920 (not shown). In addition todistances c₁ and c₂ described above, in some embodiments, processingunit 110 may further be configured to calculate distances w₁ and w₂ andmidpoint m of lane 7910 relative to one or more lane constraintsassociated with that lane. When summed together, distances w₁ and w₂equal measurement w as shown in FIG. 79D.

In this example, where a user intervenes to override the road model toalter the maneuver of the vehicle 7902 traveling in lane 7910,navigational situation information including distances c₁, c₂, d₂, w₁,and w₂ to a lane constraint 7924 may be captured and stored in memory(e.g., memory 140) and/or sent to the server for making a possibleupdate to the road model. Of course, other navigational situationinformation may also be collected and sent to a server for review. Suchinformation may include sensor outputs, captured images/image streams, aposition of the vehicle, etc. Given the permanent or semi-permanentnature of an ending lane marked by concrete barriers, the server maydecide to change or update the road model. Accordingly, vehicles mayreceive an updated to the road model that causes the vehicles to followa new or updated target trajectory for the road segment upon approachingnew lane constraint 7924.

FIG. 80 illustrates a diagrammatic side view representation of anexemplary vehicle 7902 including system 100 consistent with thedisclosed embodiments. As is shown, vehicle 7902 may be limited by avision inhibitor such as glare 8002 from the sun and/or a malfunctioninglamp 8004. Vehicle 7902 is additionally depicted with sensors 8006 andsystem 100 is capable of determining whether or not it is day or night.The sensors 8006 may include, for example, an IR sensor and/or anaccelerometer. For example, where a user intervenes to override the roadmodel to move vehicle 7902 to avoid a glare produced by sun, theprocessing unit 110 may capture navigational situation informationreflecting a time of day and/or the presence of glare. Processing unit110 may store the navigational situational information and/or transmitthe navigational situation information to a server for storage and/oranalysis. Given the temporary nature of the glare, the server may decidenot to change or update the road model.

FIG. 81 illustrates an example flowchart representing a method foradaptive navigation of a vehicle based on user intervention overridingthe road model consistent with the disclosed embodiments. In particular,FIG. 81 illustrates a process 8100 for adaptive navigation of a vehicleconsistent with disclosed embodiments. Steps of process 8100 may beperformed by processing unit 110 of system 100. Process 8100 may allowfor user input and a navigational maneuver based on analysis of anenvironmental image. Where there is user input that deviates from anavigational maneuver prescribed by the road model, the maneuver may bealtered according to the user input and the conditions surrounding theuser input may be captured and stored and/or sent to a server for makinga possible update to the road model.

At step 8110, processing unit 110 may receive at least one environmentalimage of an area forward of vehicle 7902. For example, the image mayshow one or more recognized landmarks. As discussed elsewhere in detail,a recognized landmark may be verified in the captured image and used todetermine a position of the vehicle along a target trajectory for aparticular road segment. Based on the determined position, theprocessing unit 110 may cause one or more navigational responses, forexample, to maintain the vehicle along the target trajectory.

At step 8112, processing unit 110 may include determining a navigationalmaneuver responsive to an analysis of at least one environmental imageof an area forward of vehicle 7902. For example, based on thelandmark-based position determination for the vehicle along the targettrajectory, the processing unit 110 may cause one or more navigationalresponses to maintain the vehicle along the target trajectory.

At step 8114, process 8100 may cause vehicle 7902 to initiate thenavigational maneuver. For example, processing unit 110 may sendinstructions to one or more systems associated with vehicle 7902 toinitiate the navigational maneuver and may cause vehicle 7902 to driveaccording to a predetermined trajectory along roadway 7900. Consistentwith the disclosed embodiments, an initiation instruction may be sent toa throttling system 220, braking system 230, and/or steering system 240.

At step 8116, the system may receive a user input that differs from oneor more aspects of the navigational maneuver implemented by processingunit 110 based on sparse data map 800. For example, a user input to oneor more of throttling system 220, braking system 230, and/or steeringsystem 240 may differ from an initiated maneuver and cause an overrideto alter the maneuver based on the received user input.

Based on detection of a user override or intervention condition,processing unit 110 may collect navigational situation informationrelating the vehicle and the user input at the time before, during,and/or after the user intervention. For example, processing unit 110 mayreceive information relating to the user input, including informationspecifying at least one of a degree of turn, an amount of acceleration,and an amount of braking of a vehicle 7902, etc. caused by the userintervention (step 8118).

At step 8118, processing unit 110 may determine additional navigationalsituation information relating to vehicle user input. The navigationalsituation information may include, for example, a location of thevehicle, a distance to one or more recognized landmarks, a locationdetermined by position sensor 130, one or more images captured by animage capture device of vehicle 7902, sensor outputs etc.

At step 8020, processing unit 110 may store the navigational situationinformation into memory 140 or 150 of system 100 in association withinformation relating to the user input. Alternatively, in otherembodiments, the navigational situation information may be transmittedto a server (e.g., server 1230) for use in a making a possible update tothe road model. Alternatively, in still yet other embodiments, system100 may not store the navigational situation information if system 100determines that the navigation situation information is associated witha condition that may not occur in the future (e.g., a special conditionor a transient condition), such as related to pedestrian or an animalmoving in front of vehicle 7902. System 100 may determine that suchconditions do not warrant further analysis and this may determine to notstore the navigational situation information associated with thetransient condition.

Self-Aware System for Adaptive Navigation

In some embodiments, the disclosed systems and methods may provide aself-aware system for adaptive navigation. For example, a server (e.g.,server 1230), may distribute a road model to vehicles. Based on feedbackreceived from autonomous vehicles, the system may determine whether oneor more updates (e.g., adaptations to the model) are needed to the roadmodel to account for changes in road situations. For example, in someembodiments, a vehicle (which may be an autonomous vehicle) may travelon a roadway based on the road model and may make use of observationsmade by the self-aware system in order to adjust a navigational maneuverof the vehicle based on a navigational adjustment condition. Asdiscussed herein, a navigational adjustment condition may include anyobservable or measurable condition in an environment of a vehicle. Thesystem may determine a navigational maneuver for the vehicle based, atleast in part, on a comparison of a motion of the vehicle with respectto a predetermined model representative of a road segment. The systemmay receive from a camera, at least one image representative of anenvironment of the vehicle, and then determine, based on analysis of theat least one image, an existence in the environment of the vehicle of anavigational adjustment condition. Based on this analysis, the systemmay, without user intervention, cause the vehicle to adjust thenavigational maneuver based on the existence of the navigationaladjustment condition. The system may store information relating to thenavigational adjustment condition, including, for example, data, animage, or a video related to the navigational adjustment condition. And,the system may transmit the stored information to one or moreserver-based systems for analysis and/or determination of whether anupdate to the road model is needed.

In some embodiments, the system onboard the vehicle or in the cloud mayidentify an object or a condition that is estimated to be associatedwith the navigational adjustment condition. The system may establishwhether the navigational adjustment condition is temporary or not andwhether the road model should be updated or not. The system may alsoestablish in this way whether to collect further information from futuretraversals of the same area, location, road, region, etc.

FIG. 82A illustrates a plan view of a vehicle traveling on a roadwaywith a parked car consistent with disclosed embodiments. In particular,FIG. 82A illustrates vehicle 7902 a traveling according to athree-dimensional spline representative of a predetermined path oftravel 8200 (e.g., a target trajectory) along roadway 7900 where asecond vehicle 7902 c is parked directly in front of vehicle 7902 a.Vehicle 7902 a may include a system that provides navigation features,including features that allow for navigation based on user input.Vehicle 7902 a may include components such as those discussed above inconnection with vehicle 200. For example, as depicted, vehicle 7902 amay be equipped with image capture devices 122 and 124; more or fewerimage capture devices (including cameras, for example) may be employed.

As shown, roadway 7900 may be subdivided into lanes, such as lanes 7910and 7920. Vehicle 7902 a may receive from one or more of image capturedevices 122 and 124 at least one environmental image including an imageof a parked vehicle 7902 c. In the example of FIG. 82A, vehicle 7902 ais traveling along path 8200 in lane 7910 according to instructionsderived from the road model (e.g., a heading direction along a targettrajectory) and approaching parked vehicle 7902 c. Where the systemoverrides autonomously generated steering instructions (e.g., thoseenabling the vehicle to maintain a course along the target trajectory)to adjust a maneuver of vehicle 7902 a due to a navigational adjustmentcondition, e.g., to avoid parked vehicle 7902 c, navigational adjustmentcondition information may be captured and stored in memory (e.g., memory140) and/or sent to a server (e.g., server 1230) for making a possibleupdate to the road model. In this example, the navigational adjustmentcondition information may include a location of vehicle 7902 e when theautonomous navigational change (e.g., made by the self-aware system) wasmade. The vehicle position may be identified by position sensor 130 orbased on a landmark-based determination of position along a targettrajectory. Other navigational condition information may be included inone or more images captured by an image capture device included invehicle 7902 c depicting the vehicle's environment (e.g., an imageincluding parked vehicle 7902 c), an image stream (e.g., a video),and/or sensor output data (e.g., from speedometers, accelerometers,etc.).

In some embodiments, processing unit 110 may send the navigationalsituational information from the vehicle to the server via a wirelessdata connection over one or more networks (e.g., over a cellular networkand/or the Internet, etc.). The server side may analyze the receivedinformation (e.g., using automated image analysis processes) todetermine whether any updates to sparse data model 800 are warrantedbased on the detected system intervention. In this example, the servermay recognize the presence of the parked car in or near a targettrajectory of the host vehicle and determine that the parked carrepresents a temporary or transient condition. Therefore, the server maydetermine not to change or update the road model. However, in someembodiments, based on the location of vehicle 7902 a, the server maydetermine that the parked car is located in a residential area andtherefore may change or update the road model due to the likelihood ofvehicles being parked along the shoulder of the road. Furthermore, insome embodiments, system 100 onboard the vehicle may classify an objector condition and system 100 may determine whether or not to change orupdate the road model.

FIG. 82B illustrates a plan view of a vehicle traveling on a roadwayalong a target trajectory associated with the road segment consistentwith the disclosed embodiments. Vehicle 7902 may receive from imagecapture devices 122 and 124 at least one environmental image of aturning roadway 7900 representative of a lane 7910 ending. This changein lane 7910 may be due to recent modifications to a road, and thus maynot be yet reflected in the sparse data model 800.

In this example, the vehicle systems may recognize the ending lane andoverride navigation according to the road model in order to adjust amaneuver of the vehicle 7902 traveling along path 8200 in lane 7910. Forexample, processing unit 110, using one or more images captured withcameras aboard the vehicle may recognize a blockage in the path alongthe target trajectory associated with the road segment. Processing unit110 may adjust steering of the vehicle to leave a path indicated by thetarget trajectory in order to avoid lane constraint 7924. As a result ofthe system generated navigational adjustment, navigational adjustmentcondition information (e.g., including the existence of an ending oflane 7910, any of distances c₁, c₂, d₂, r, w₁, and w₂ etc.) may bestored in memory (e.g., memory 140) and/or sent to a server (e.g. server123) for possible update of the road model. In some embodiments, inaddition or alternatively, the navigational adjustment conditioninformation may include a location of vehicle 7902 based on datadetermined by position sensor 130 and/or a position of vehicle 7902relative to one or more recognized landmarks.

The server side may analyze the received information (e.g., usingautomated image analysis processes) to determine whether any updates tosparse data model 800 are warranted based on the detected systemintervention. In some embodiments, the server may or may not update theroad model based on the received navigational adjustment conditioninformation. For example, given the permanent nature of an ending laneaccompanied by a lane shift, the server may decide it is necessary tochange or update the road model. Accordingly, the sever may modify theroad model in order to steer or turn to merge at these distances c₁, c₂,d₂, w₁, and w₂ upon approaching lane constraint 7924. The model may alsobe updated based on a received, reconstructed and actual trajectorytaken by vehicle 7902 as it navigated past the ending lane.Additionally, rather than aggregating the actual path of vehicle 7902with other trajectories stored in sparse data model 800 for theparticular road segment (e.g., by averaging the path of vehicle 7902with other trajectories stored in sparse data model 800), the targettrajectory may be defaulted to the path of vehicle 7902. That is,because the server may determine that the cause of the navigationalchange was a non-transient (or semi-permanent) condition, the path ofvehicle 7902 may be more accurate for the particular road segment thanother trajectories for the road segment collected before the conditionexisted. The same approach and analysis could also be employed by theserver upon receiving navigational modifications based not on control bythe self-aware vehicle system, but on user intervention (describedabove).

FIG. 82C illustrates a plan view of a vehicle traveling on a roadwayapproaching a pedestrian consistent with disclosed embodiments. In theexample of FIG. 82C, vehicle 7902 is driving in lane 7910 of roadway7900 with pedestrian 7926. As shown, pedestrian 7926 may be positioneddirectly in roadway 7900 or alternatively may be positioned to the sideof roadway 7900. Vehicle 7902 may travel in lane 7910 according toinstructions derived based on the road model (e.g., a heading directionalong a target trajectory) and may approach pedestrian 7926. Vehicle7902 may receive from image capture devices 122 and 124 at least oneenvironmental image including an image of pedestrian 7926. Where thesystem intervenes to override the road model to adjust a maneuver of thevehicle 7902 traveling in lane 7910 to avoid pedestrian 7926,navigational adjustment condition information including, for example, adistance d₁ to a stop sign and/or a capture image depicting pedestrian7926 may be captured and stored in memory (e.g., memory 140) and/or sentto a server (e.g., server 1230) for making a possible update to the roadmodel. The server side may analyze the received information (e.g., usingautomated image analysis processes) to determine whether any updates tosparse data model 800 are warranted based on the detected systemintervention. In this example, given the temporary nature of apedestrian, the server may determine to not change or update the roadmodel.

Optionally, in some embodiments, when the cause of the intervention isnot confidently ascertained by the system, or when the nature of thecause in not clear or is inherently not constant or stable, the servermay issue an alert and/or provide one, two, or more alternative paths orroad models. In such an embodiment, the server may cause the systemonboard the vehicle to examine the situation on the ground includingwhen the vehicle arrived at the point or area where the deviation orintervention occurred. The server may further provide a location of asuspected and/or verified cause of the intervention, to allow the systemto focus on that area. As such, the system may have more time and moreinformation to evaluate the situation.

FIG. 82D illustrates a plan view of a vehicle traveling on a roadwayapproaching an area of construction consistent with the disclosedembodiments. As shown, vehicle 7902 is traveling a target trajectoryassociated with the road segment (e.g., according to a three-dimensionalspline representative of a predetermined target trajectory 8200) along aroadway 7900 where a construction area 8200 d is located directly infront of vehicle 7902. Vehicle 7902 may receive from image capturedevices 122 and 124 at least one environmental image including an imageof construction area 8200 d. Where the system intervenes to override oneor more navigational maneuvers generated based on the road model inorder to avoid construction area 8200 d, navigational adjustmentcondition information may be stored. Such information may include, forexample, the existence of a construction area 8200 d (e.g., as depictedin one or more captured images). The navigational adjustment conditioninformation may also be sent to a server (e.g. server 120) for a makingone or more possible updates to sparse data model 800. In someembodiments, the navigational adjustment condition information mayinclude a location of vehicle 7902 based on, for example, a positionsensor 130 and/or a location of a known landmark relative to vehicle7902 at the time of adjustment. The server side may analyze the receivedinformation (e.g., using automated image analysis processes) todetermine whether any updates to sparse data model 800 are warrantedbased on the detected system intervention. In this example, due to thenon-transient nature of the roadway construction (where non-transientmay refer to a condition likely to exist longer than a predeterminedperiod of time, including, for example, several hours, a day, a week, amonth, or more), the server may determine to change or update the roadmodel.

FIG. 83 illustrates an example flowchart representing a method for modeladaptation based on self-aware navigation of a vehicle consistent withthe disclosed embodiments. In particular, FIG. 83 illustrates a process8300 that may be performed by processing unit 110 of system 100. Asdiscussed below, process 8300 may use a road model defining apredetermined vehicle trajectory 8200. Where a maneuver deviates fromnavigational maneuvers developed based on the predetermined modelvehicle trajectory 8200, the model and information regarding anavigational adjustment condition may be captured and stored and/or sentto a server (e.g., server 1230) for making a possible update to the roadmodel.

At step 8310, processing unit 110 may determine a navigational maneuverbased on a comparison of a vehicle position with respect to apredetermined model associated with a road segment. As discussedelsewhere in detail, a recognized landmark may be verified in thecaptured image and used to determine a position of the vehicle along atarget trajectory for a particular road segment. Based on the determinedposition, the processing unit 110 may cause one or more navigationalresponses, for example, a navigational maneuver to maintain the vehicle(e.g., steer the vehicle) along the target trajectory.

At step 8312, processing unit 110 may receive an environmental image ofan area forward of vehicle 7902. For example, processing unit 110 mayreceive an image of an environment of vehicle 7902 that includes aparked vehicle, a lane constraint having a road curvature or turningroadway radius r providing information indicative, for example, of aroadway lane ending, a pedestrian and/or construction area.

At step 8314, processing unit 110 may determine an existence of anavigational adjustment condition. The navigational adjustment conditionmay be determined responsive to an analysis of at least oneenvironmental image of an area forward of vehicle 7902 and may include,for example, a parked car in front of vehicle 7902 a, a roadwaycurvature having turn radius r providing information indicative, forexample, of construction area in lane 7910. These are, of course,examples, and the captured images may include any of a multitude ofconditions within an environment of the vehicle that may warrant anadjustment in navigation away from a target trajectory included insparse data model 800.

At step 8316, processing unit 110 may cause vehicle 7902 to adjust thenavigational maneuver based on the navigational adjustment condition.For example, processing unit 110 may cause vehicle 7902 to changeheading directions away from a direction of the target trajectory inorder to avoid a parked car, a road construction site, a pedestrian,etc. Consistent with the disclosed embodiments, instructions may be sentto a throttling system 220, braking system 230, and/or steering system240 in order to cause the adjustment to one or more navigationalmaneuvers generated based on sparse data model 800.

At step 8318, processing unit 110 may store information relating to thenavigational adjustment condition information into memory 140 or 150 ofsystem 100. Such information may include one or more images captured ofthe environment of the vehicle at the time of the navigationaladjustment that resulted in a departure from the target trajectory ofsparse data model 800. The information may also include a position ofthe vehicle, outputs of one or more sensors associated with the vehicle,etc.

At step 8320, processing unit 110 may transmit the navigationaladjustment condition information to a road model management system(e.g., server 1230) for analysis and for potentially updating apredetermined model representative of the roadway.

Adaptive Road Model Manager

In some embodiments, the disclosed systems and methods may provide anadaptive road model manager. The adaptive road model manager may beprovided by a server (e.g., server 1230), which may receive data fromvehicles and decide whether or not to make an update to the road modelif an adjustment from an expected vehicular navigational maneuver wasnot due to a transient condition. The vehicles may send data to theserver regarding navigational departures from the road model using awireless data connection over one or more networks (e.g., including overa cellular network and/or the Internet). For example, the server mayreceive from each of a plurality of autonomous vehicles navigationalsituation information associated with an occurrence of an adjustment toa determined navigational maneuver. The server may analyze thenavigational situation information and determine, based on the analysisof the navigational situation information, whether the adjustment to thedetermined navigational maneuver was due to a transient condition. Insome embodiments, the server may detect the navigational maneuver fromraw data provided by the vehicle (e.g., by processing image data). Theserver may update the predetermined model representative of the at leastone road segment if the adjustment to the determined navigationalmaneuver was not due to a transient condition. As discussed herein, atransient condition is any condition expected to change after apredetermined time period (e.g., less than a few hours, a day, or a weekor more) such that an update to a road model is not warranted ordesirable. Such transient conditions may be expected to no longer bepresent after the predetermined time period and therefore the server maydetermine to not change or update to the road model. Conversely, if theserver determines the adjustment was not due to a transient condition,the server may determine to update the road model.

In some embodiments, when a navigational maneuver is detected, theserver may mark the respective area of the road model as beingassociated with a suspected change. The server may then determine fromfurther updates from the same location or a nearby location (e.g., insome embodiments, “'pulling” such updates from vehicles at the locationor nearby the location), and may process the data in an attempt toverify the change. When the change is verified, the server may updatethe model, and may subsequently communicate the updated model of therespective area, replacing the former version of the model. The servermay implement a confidence level such that the update occurs when theconfidence level is above a certain level. The confidence level may beassociated with the type of maneuver, the similarity between two or moremaneuvers, identification of a source of the adjustment, a frequency ofconsistent updates, and the number of ratio of inconsistent updates,environmental conditions, such as weather, urban vs. rural environments,etc. The severity of the cause of the maneuver may also be taken intoaccount when determining the confidence level. If the maneuver is severe(e.g., a sharp turn) and the cause may be associated with a potentialweather situation and, in some embodiments, a less restrictive approvalprocess can may be used.

FIG. 84A illustrates a plan view of a vehicle traveling on a roadwaywith multiple parked cars consistent with the disclosed embodiments. Asshown, vehicle 7902 a is traveling according to a target trajectory(e.g., a three-dimensional spline representative of a predetermined pathof travel 8400) of a road model along roadway 7900 where another vehicle7902 c is parked directly in front of vehicle 7902 a. Roadway 7900 maybe subdivided into lanes, such as lanes 7910 and 7920. Where either thesystem or user intervenes to override a navigational maneuver generatedbased on the road model and adjust a maneuver of the vehicle 7902traveling along path 8400 in lane 7910 to avoid parked vehicles 7902 c,navigational situation information including, for example, the existenceof parked cars 7902 c in lane 7910 (e.g., as depicted in one or moreimages captured by an image captured device of vehicle 7902 a) may bemay be sent to a server (e.g., server 1230) for analysis.

The server side may analyze the received information (e.g., usingautomated image analysis processes) to determine whether any updates tosparse data model 800 are warranted based on whether or not theadjustment was due to a transient condition. Where the adjustment wasnot due to the existence of a transient condition, the road model may beupdated. For example, where an experienced condition is determined to beone likely to persist beyond a predetermined time threshold (e.g., a fewhours, a day, or a week or more) updates may be made to the model. Insome embodiments, the threshold for determining a transient conditionmay be dependent on a geographic region in which the condition isdetermined to occur, on an average number of vehicles that travel theroad segment in which the condition was encountered, or any othersuitable criteria. For example, in geographic regions, such as ruralregions, that include fewer vehicles likely to encounter a road-relatedcondition, a time threshold for making the transient or not transientdetermination may be longer that another geographic region (e.g., anurban environment) that includes more vehicles likely to encounter theroad-related condition over a particular time period. That is, as theaverage number of vehicles traveling a road segment increases, the timethreshold for making the transient determination may be lower. Such anapproach may reduce the number of cars traveling in an urban environmentthat will need to rely upon their internal systems (camera, sensors,processor, etc.) to recognize a road condition that warrants anavigational response different from one expected based on sparse model800. At the same time, a longer transient time threshold in lowertrafficked areas may reduce the likelihood that the model is changed toaccount for an experienced road condition and, a short time later (e.g.,within hours, a day, etc.) needs to be changed back to its originalstate, for example, after the experience road condition no longerexists.

Conversely, where a navigational adjustment is determined to be inresponse to a transient condition, the server may elect to not make anyupdates to the road model. For example, where either the system or userintervenes to navigate the vehicle 7902 a into lane 7920 to avoidvehicles 7902 c parked on the shoulder yet abutting into lane 7910 (FIG.84A), where ith the system or user navigates the host vehicle to avoidan intervening car 7902 d (FIG. 84B), wherein the system or usernavigates the host vehicle to avoid a temporary barrier 8402 (such as afallen tree, as shown in FIG. 84C), or where the system or usernavigates the host vehicle to avoid markers 8200 d designating temporaryroadwork (FIG. 84D), where the system or user navigates the host vehicleto avoid a pothole 8502 present in the roadway (FIG. 85A), where thesystem or user navigates the host vehicle to avoid a pedestrian 8504 orpedestrian in the roadway (FIG. 85B), the server may determine in eachcase that the experienced condition constitutes a transient conditionnot warranting an update to sparse data model 800.

In some cases, and as described above, certain road conditions may beclassified as transient based on a determination of a probable time oftheir existence (less than a few hours, a day, a week, etc.). In othercases, a determination of whether a certain road condition is atransient one may be based on factors other than or in addition to time.For example, in the case of a pothole captured in one or more images,the server (or the processing unit associated with a host vehicle) maydetermine a depth of the pothole, which may aid in determining whetherthe pothole represents a transient condition and, therefore, whethersparse data model 800 should be updated in view of the pothole. If thepothole 8502 is determined to have a depth that could result inpotential damage to the host vehicle if driven through (e.g., a depth onthe order of greater than 3 cm, 5 cm, 10 cm or more), then the potholemay be categorized as non-transient. Similarly, if the pothole 8502 islocated in a geographic region in which road repair is known to besomewhat slow (e.g., requiring more than a day to repair, a week torepair, or longer), then a pothole may be categorized as non-transient.

Determination of whether a particular road condition constitutes atransient condition may be fully automated and performed by one or moreserver-based systems. For example, in some embodiments, the one or moreserver based systems may employ automated image analysis techniquesbased on one or more images captured by cameras onboard a host vehicle.In some embodiments, the image analysis techniques may include machinelearning systems trained to recognize certain shapes, road features,and/or objects. For example, the server may be trained to recognized inan image or image stream the presence of a concrete barrier (possiblyindicating the presence of a non-transient construction or laneseparation condition), a pothole in the surface of the road (a possibletransient or non-transient condition depending on the size, depth,etc.), a road edge intersecting with an expected path of travel(potentially indicating a non-transient lane shift or new trafficpattern), a parked car (a potentially transient condition), an animalshape in the road (a potentially transient condition), or any otherrelevant shapes, objects, or road features.

The image analysis techniques employed by the server may also include atext recognition component to determine a meaning associated with textpresent in an image. For example, where text appears in one or moreuploaded images from an environment of a host vehicle, the server maydetermine whether text exists in the images. If text exists, the servermay use techniques such as optical character recognition to assist indetermining whether the text may relate to a reason that a system oruser of a host vehicle caused a navigational maneuver differing fromthat expected based on sparse model 800. For example, where a sign isidentified in an image, and the sign is determined to include the text“NEW TRAFFIC PATTERN AHEAD,” the text may assist the server indetermining that the experienced condition had a non-transient nature.Similarly, signs such as “ROAD CLOSED AHEAD” or “BRIDGE OUT” may alsohelp indicate the presence of a non-transient condition for which anupdate to sparse road model 800 may be justified.

The server-based system may also be configured to take into accountother information when determining whether an experienced road conditionis transient. For example, the server may determine an average number ofvehicles that travel a road segment over a particular amount of time.Such information may be helpful in determining the number of vehicles atemporary condition is likely to affect over an amount of time that thecondition is expected to persist. Higher numbers of vehicles impacted bythe condition may suggest a determination that the sparse data model 800should be updated.

In some embodiments, determination of whether a particular roadcondition constitutes a transient condition may include at least somelevel of human assistance. For example, in addition to the automatedfeatures described above, a human operator may also be involved inreviewing information uploaded from one or more vehicles and/ordetermining whether sparse data model 800 should be updated in view ofthe received information.

FIG. 86 illustrates an example flowchart representing a method for anadaptive road model manager consistent with disclosed embodiments. Inparticular, FIG. 86 illustrates a process 8600 for an adaptive roadmodel manager consistent with disclosed embodiments. Steps of process8600 may be performed by a server (e.g., server 1230), which may receivedata from a plurality of autonomous vehicles over one or more networks(e.g., cellular and/or the Internet, etc.).

At step 8610, the server may receive from each of a plurality ofautonomous vehicles navigational situation information associated withan occurrence of an adjustment to a determined navigational maneuver.The navigational situation information may result from system or userinvention overriding the road model. The navigational situationinformation may include at least one image or a video representing anenvironment of vehicle 7902. In some embodiments, the navigationalsituation information may further include a location of vehicle 7902(e.g., as determined by position sensor 130 and/or based on a distanceof vehicle 7902 to a recognized landmark).

At step 8612, the server may analyze the navigational situationinformation. For example, the server side may analyze the receivedinformation (e.g., using automated image analysis processes) todetermine what is depicted in the at least one image or videorepresenting an environment of vehicle 7902. This analysis may includeidentification of the existence of, for example, a parked car, anintervening car, a temporary barrier, such as a fallen tree directly infront of a vehicle, roadwork, a low light condition, a glare condition,a pothole, an animal, or a pedestrian.

At step 8614, the server may determine, based on the analysis of thenavigational situation information, whether the adjustment to thedetermined maneuver was due to a transient condition. For example, Atransient condition may include where a second vehicle is parkeddirectly in front of a vehicle, a vehicle intervenes directly in frontof vehicle, barrier, such as a fallen tree lies directly in front of avehicle, a low light condition, a glare condition, a pothole (e.g., oneof a minimal depth), an animal, or a pedestrian.

At step 8616, process 8600 may include the server updating thepredetermined model representative of the at least one road segment ifthe adjustment to the determined navigational maneuver was not due to atransient condition. For example, a condition that may be non-transientmay include a substantial pothole, long-term and/or extensive roadwork,etc. This update may include an update to the three-dimensional splinerepresenting a predetermined path of travel along at least one roadsegment.

Road Model Management Based on Selective Feedback

In some embodiments, the disclosed systems and methods may manage a roadmodel based on selective feedback received from one or more vehicles. Asdiscussed in earlier sections, the road model may include a targettrajectory (e.g., a three-dimensional spline representing apredetermined path of travel along a road segment). Consistent withdisclosed embodiments, a server (e.g., server 1230) may selectivelyreceive road environment information from autonomous vehicles in orderto update the road model. As used herein, road environment informationmay include any information related to an observable or measurablecondition associated with a road or a road segment. The server mayselectively receive the road environment information based on a varietyof criteria. Relative to the disclosed embodiments, selectivelyreceiving information may refer to any ability of a server based systemto limit data transmissions sent from one or more autonomous vehicles tothe server. Such limitations placed on data transmissions from the oneor more autonomous vehicles may be made based any suitable criteria.

For example, in some embodiments, the server may limit a frequency atwhich road environment information is uploaded to the server from aparticular vehicle, from a group of vehicles, and/or from vehiclestraveling within a particular geographic region. Such limitations may beplaced based on a determined model confidence level associated with aparticular geographic region. In some embodiments, the server may limitdata transmissions from autonomous vehicles to only those transmissionsincluding information suggesting a potential discrepancy with respect toat least one aspect of the road model (such information, for example,may be determined as prompting one or more updates to the model). Theserver may determine whether one or more updates to the road model arerequired based on the road environment information selectively receivedfrom the autonomous vehicles and may update the road model to includethe one or more updates. Examples of a server selectively receiving roadenvironment information from autonomous vehicles are discussed below.

FIG. 87A illustrates a plan view of a vehicle traveling on an interstateroadway consistent with the disclosed embodiments. As shown, vehicle7902 is traveling along a predetermined path of travel 8700 (e.g., atarget trajectory according to a road model) associated with interstateroadway 7900. As shown, roadway 7900 may be subdivided into lanes, suchas lanes 7910 and 7920. The server may selectively receive roadenvironment information based on navigation by vehicle 7902 through aroad environment, such as roadway 7900. For example, the roadenvironment information may include one or more images captured by animage capture device of vehicle 7902; location information representinga position of vehicle 7902 determined by, for example, using positionsensor 130 and/or based on a position of vehicle 7902 relative to arecognized landmark; outputs from one or more sensors associate withvehicle 7902, etc. Based upon the road environment information, theserver may determine whether updates to the road model are required.

In the example shown in FIG. 87A, a single particular vehicle 7902 isshown traveling along an interstate roadway 7900 and following a targettrajectory 8700. FIG. 87B illustrates a plan view of a group of vehicles7902 e, 7902 f, 7902 g, and 7902 h traveling along a city roadway 7900and following target trajectories 8700 a and 8700 b that may beassociated with lanes 7910 and 7920 of roadway 7900, for example. FIG.87C illustrates a plan view of a vehicle 7902 i traveling within a ruralgeographic region 8722 on roadway 7900. FIG. 87D illustrates a vehicle7902 traveling on a roadway 7900 including a newly modified trafficpattern. For example, where once lane 7910 may have extended forward ofvehicle 7902, a new traffic pattern may exist where lane 7910 now comesto an end forward of vehicle 7902.

Information relating to the navigation of vehicle 7902 in any of thesesituations, among others, may be collected and uploaded to one or moreserver based systems that maintain sparse data map 800. Based on thereceived information, the server may analyze whether one or more updatesare needed to sparse data map 800 and, if an update is determined to bejustified, then the server may make the update to sparse data map 800.In some embodiments, the analysis and updating may be performedautomatically by the server via automated image analysis of imagescaptured by cameras aboard vehicle 7910, automated review of sensor andposition information, automated cross-correlation of informationreceived from multiple autonomous vehicles, etc. In some embodiments, anoperator associated with the server-based system may assist in review ofthe information received from the autonomous vehicles and determinationof whether updates to sparse data model 800 are needed based on thereceived information.

In some embodiments, the server may be configured to receivenavigational information from all available autonomous vehicles.Further, this information may be uploaded to the server based on apredetermined protocol. For example, the information may be uploadedacross a streaming data feed. Additionally or alternatively, theinformation may be uploaded to the server at a predetermined periodicrate (e.g., several times per second, once per second, once per minute,once every several minutes, once per hour, or any other suitable timeinterval). The information may also be uploaded to the server based onaspects of the vehicle's navigation. For example, navigationalinformation may be uploaded from a vehicle to the server as the vehiclemoves from one road segment to another or as the vehicle moves from onelocal map associated with sparse data map 800 to another.

In some embodiments, the server may be configured to selectively controlthe receipt of navigational information from one or more autonomousvehicle. That is, rather than receiving all available navigationalinformation from all available autonomous vehicles, the server mayrestrict the amount of information it receives from one or moreavailable autonomous vehicles. In this way, the server may reduce theamount of bandwidth needed for communicating with available autonomousvehicles. Such selective control of information flow from the autonomousvehicles and the server may also reduce an amount of processingresources required to process the communications incoming from theautonomous vehicles.

The selective control of information flow between the autonomousvehicles and the server may be based on any suitable criteria. In someembodiments, the selectivity may be based on the type of road that avehicle is traversing. With reference to the example shown in FIG. 87A,vehicle 7902 is traversing an interstate, which may be a well-traveledroad. In such situations, the server may have accumulated a significantamount of navigational information relating to the interstate road, itsvarious lanes, the landmarks associated with the road, etc. In suchcircumstances, continuing to receive full information uploads from everyvehicle that travels along the interstate roadway may not contribute tosignificant or further refinements of the road model represented insparse data map 800. Therefore, the server may limit, or an autonomousvehicle traveling along a certain type of road or a particular roadsegment may limit, the amount or type of information uploaded to theserver.

In some embodiments, the server may forego automatic information uploadsaltogether from vehicles traveling along a particular interstateroadway, a heavily traveled urban road, or any other road where sparsedata model 800 is determined to require no additional refinements.Instead, in some embodiments, the server may selectively acquire datafrom vehicles traveling along such roads as a means for periodicallyconfirming that sparse data map 800 remains valid along selectedroadways. For example, the server may interrogate one or more vehiclesdetermined to be traveling along an interstate, heavily traveled roadsegment, etc. to collect navigational information from the interrogatedvehicle. This information may include information relating to areconstructed trajectory of the vehicle along the roadway, a position ofthe vehicle on the roadway, sensor information from the vehicle,captured images from cameras onboard the vehicle, etc. Using thistechnique, the server may periodically monitor the state of a roadwayand determine whether updates are needed to sparse data model 800without unnecessary usage of data transmission and/or data processingresources.

In some embodiments, the server may also selectively control data flowfrom an autonomous vehicle based on the number of cars determined to betraveling within a group along a roadway. For example, where a group ofautonomous vehicles (e.g., two or more vehicles) is determined to betraveling within a certain proximity of one another (e.g., within 100meters, 1 km, or any other suitable proximity envelope), informationupload may be restricted from any of the members of the group. Forexample, the server may restrict information transfer to only one memberof the group, any subset of members of the group, one member of thegroup from each lane of the road, etc.

In some embodiments, the server may also selectively control data flowfrom an autonomous vehicle based on a geographic region. For example,some geographic regions may include road segments for which sparse datamodel 800 already includes refined target trajectories, landmarkrepresentations, landmark positions, etc. For example, in certaingeographic regions (e.g., urban environments, heavily traveled roadways,etc.), sparse data model 800 may be generated based upon multipletraversals of various road segments by vehicles in a data collectionmode. Each traversal may result in additional data relevant to roadsegments in a geographic region from which sparse data model 800 may berefined. In some cases, sparse data map 800 for certain geographicregions may be based upon 100, 1000, 10000 or more prior traversals ofvarious road segments. In those regions, additional information receivedfrom one or more autonomous vehicles may not serve as a basis forfurther, significant refinements of sparse data model. Thus, the servermay restrict uploads from vehicles traveling in certain geographicregions. For example, in some cases, the server may preclude allautomatic transmissions of road data from vehicles traveling in selectedgeographic regions. In other cases, the server may enable transmissionof data from only a portion of vehicles traveling in a certaingeographic region (e.g., 1 of 2 vehicles, 1 of 5, 1 of 100, etc.). Inother cases, the server may receive transmissions from only thosevehicles in a geographic location that the server identifies and queriesfor updated road information. The server can use information receivedfrom any portion of the vehicles from a certain geographic region toverify and/or update any aspect of sparse data model 800.

In some embodiments, the server may also selectively control data flowfrom an autonomous vehicle based on a confidence level assigned to aparticular local map, road segment, geographic region, etc. For example,like the geographic region example, certain road segments, local maps,and/or geographic regions may be associated with a confidence levelindicative of, for example, a level of refinement of sparse data map 800in those areas. The server may restrict transmission of road informationfrom vehicles traveling on any roads, local map areas, or geographicregions associated with a confidence level above a predeterminedthreshold. For example, in some cases, the server may preclude allautomatic transmissions of road data from vehicles traveling in regionswith a confidence level above a predetermined threshold. In other cases,the server may enable transmission of data from only a portion ofvehicles traveling in those regions (e.g., 1 of 2 vehicles, 1 of 5, 1 of100, etc.). In other cases, the server may receive transmissions fromonly those vehicles in a high-confidence area (one including aconfidence level above a predetermined threshold) that the serveridentifies and queries for updated road information. The server can useinformation received from any portion of the vehicles from ahigh-confidence level region to verify and/or update any aspect ofsparse data model 800.

In some embodiments, the server may also selectively control data flowfrom an autonomous vehicle based on the type of information includedwithin the navigational information to be uploaded by a particularautonomous vehicle. For example, in many cases, the road informationuploaded to the server from various host vehicles may not significantlyimpact sparse data model 800. For example, in high-confidence levelgeographic areas or road segments etc., additional road information fromtraversing vehicles may be useful for verifying the continued accuracyof sparse data model 800, but such information may not offer a potentialfor additional significant refinements to sparse data model 800. Thus,continued transmission of information that verifies sparse data model800, but does not offer a potential for significant further refinementof sparse data model 800 may consume data transmission and processingresources without a potential for significant benefit.

In such cases, it may be desirable for the server to limit datatransmissions from vehicles. Instead of receiving data transmissionsautomatically from all (or even a part of) available vehicles, theserver may restrict data transmissions from vehicles to only thoseexperiencing situations that may impact sparse road model 800. Forexample, where a vehicle traversing a road segment experiences asituation that requires a navigational response that departs from oneanticipated by the sparse data model 800 (e.g., where the vehicle musttravel a path different from a target trajectory for a road segment),then the processing unit 110 may determine that such a departure hasoccurred and may relay that information to the server. In response, theserver may query the vehicle for information relating to thenavigational departure so that the server can determine whether anyupdates are needed to sparse data model 800. In other words, the servermay elect to receive road information from vehicles only where theinformation suggests that a change may be needed to sparse data model800.

FIG. 88 illustrates an example flowchart representing a method for roadmodel management based on selective feedback consistent with thedisclosed embodiments. Steps of process 8800 may be performed by aeserver (e.g., server 1230). As discussed below, process 8800 may involveselectively receiving feedback to potentially update the road modelbased upon road environment information from autonomous vehicles.

At step 8810, the server may selectively receive road environmentinformation based on navigation from a plurality of autonomous vehiclesthrough their respective road environments. For example, the server mayselectively apply a limitation on a frequency of informationtransmissions received from a particular vehicle, from a group ofvehicles, from vehicles traveling within a particular geographic region,or from vehicles based on a determined model confidence level associatedwith a particular geographic region. Further, in some embodiments, theserver may selectively limit data transmissions from vehicles only tothose transmissions that reflect a potential discrepancy with respect toat least one aspect of a predetermined road model.

At step 8812, the server may determine whether one or more updates tothe road model are required based on the road environment information.If the server determines that updates to the road model are justifiedbased on information selectively received from one or more autonomousvehicles, those updates may be made at step 8814.

The foregoing description has been presented for purposes ofillustration. It is not exhaustive and is not limited to the preciseforms or embodiments disclosed. Modifications and adaptations will beapparent to those skilled in the art from consideration of thespecification and practice of the disclosed embodiments. Additionally,although aspects of the disclosed embodiments are described as beingstored in memory, one skilled in the art will appreciate that theseaspects can also be stored on other types of computer readable media,such as secondary storage devices, for example, hard disks or CD ROM, orother forms of RAM or ROM, USB media, DVD, Blu-ray, 4K Ultra HD Blu-ray,or other optical drive media.

Computer programs based on the written description and disclosed methodsare within the skill of an experienced developer. The various programsor program modules can be created using any of the techniques known toone skilled in the art or can be designed in connection with existingsoftware. For example, program sections or program modules can bedesigned in or by means of .Net Framework, .Net Compact Framework (andrelated languages, such as Visual Basic, C, etc.), Java, C++,Objective-C, HTML, HTML/AJAX combinations, XML, or HTML with includedJava applets.

Moreover, while illustrative embodiments have been described herein, thescope of any and all embodiments having equivalent elements,modifications, omissions, combinations (e.g., of aspects across variousembodiments), adaptations and/or alterations as would be appreciated bythose skilled in the art based on the present disclosure. Thelimitations in the claims are to be interpreted broadly based on thelanguage employed in the claims and not limited to examples described inthe present specification or during the prosecution of the application.The examples are to be construed as non-exclusive. Furthermore, thesteps of the disclosed methods may be modified in any manner, includingby reordering steps and/or inserting or deleting steps. It is intended,therefore, that the specification and examples be considered asillustrative only, with a true scope and spirit being indicated by thefollowing claims and their full scope of equivalents.

What is claimed is:
 1. A system for navigating a vehicle on a road atleast partially covered with snow, the system comprising: at least oneprocessor programmed to: receive from an image capture device, aplurality of images captured of an environment forward of the vehicle,including an area where snow covers a road on which the vehicle travels;analyze at least one of the plurality of images to identify a first freespace boundary on a first side of the vehicle and extending forward ofthe vehicle, a second free space boundary on a second side of thevehicle and extending forward of the vehicle, and a forward free spaceboundary forward of the vehicle and extending between the first freespace boundary and the second free space boundary, wherein the firstfree space boundary, the second free space boundary, and the forwardfree space boundary define a free space region forward of the vehicle;determine a first proposed navigational path for the vehicle through thefree space region; provide the at least one of the plurality of imagesto a neural network and receive from the neural network a secondproposed navigational path for the vehicle based on analysis of the atleast one of the plurality of images by the neural network; determinewhether the first proposed navigational path agrees with the secondproposed navigational path; and cause the vehicle to travel on at leasta portion of the first proposed navigational path if the first proposednavigational path is determined to agree with the second proposednavigational path.
 2. The system of claim 1, wherein analyzing the atleast one image includes identifying at least one tire track in thesnow.
 3. The system of claim 1, wherein analyzing the at least one imageincludes identifying a change of light across a surface of the snow. 4.The system of claim 1, wherein analyzing the at least one image includesidentifying a plurality of trees, a barrier, a row of parked cars, acurb, or a snow bank along an edge of the road.
 5. The system of claim1, wherein analyzing the at least one image includes recognizing achange in curvature at a surface of the snow.
 6. The system of claim 5,wherein the recognized change in curvature is determined to correspondto a probable location of a road edge.
 7. The system of claim 1, whereinanalyzing the at least one image includes a pixel analysis of the atleast one image in which at least a first pixel is compared to at leasta second pixel in order to determine a feature associated with a surfaceof the snow covering at least some lane markings and road edges.
 8. Thesystem of claim 7, wherein the feature corresponds to an edge of a tiretrack.
 9. The system of claim 7, wherein the feature corresponds to anedge of the road.
 10. The system of claim 7, wherein the at least oneprocessor is further programmed to cause the vehicle to navigate alongthe first proposed navigational path between determined edges of theroad.
 11. The system of claim 7, wherein the at least one processor isfurther programmed to cause the vehicle to navigate along the firstproposed navigational path by at least partially following tire tracksin the snow.
 12. The system of claim 1, wherein the at least oneprocessor is further programmed to cause the vehicle to determine thefirst proposed navigational path based at least on a width of the freespace region.
 13. The system of claim 1, wherein the at least oneprocessor is further programmed to: cause the vehicle to detect in atleast one image a target vehicle; and determine a lateral position ofthe target vehicle.
 14. The system of claim 1, wherein the at least oneprocessor is further programmed to determine the first proposednavigational path based on a position of the vehicle.
 15. The system ofclaim 1, wherein the at least one processor is further programmed to:determine a lateral position of a target relative to the first proposednavigational path; and validate or refine the first proposednavigational path.
 16. The system of claim 1, wherein the first proposednavigational path agrees with the second proposed navigational path whena deviation between the first proposed navigational path and the secondproposed navigational path is less than a certain threshold.
 17. Amethod for navigating a vehicle on a road at least partially coveredwith snow, the method comprising: receiving from an image capturedevice, a plurality of images captured of an environment forward of thevehicle, including an area where snow covers a road on which the vehicletravels; analyzing at least one of the plurality of images to identify afirst free space boundary on a first side of the vehicle and extendingforward of the vehicle, a second free space boundary on a second side ofthe vehicle and extending forward of the vehicle, and a forward freespace boundary forward of the vehicle and extending between the firstfree space boundary and the second free space boundary, wherein thefirst free space boundary, the second free space boundary, and theforward free space boundary define a free space region forward of thevehicle; determining a first proposed navigational path for the vehiclethrough the free space region; providing the at least one of theplurality of images to a neural network and receive from the neuralnetwork a second proposed navigational path for the vehicle based onanalysis of the at least one of the plurality of images by the neuralnetwork; determining whether the first proposed navigational path agreeswith the second proposed navigational path; and causing the vehicle totravel on at least a portion of the first proposed navigational path ifthe first proposed navigational path is determined to agree with thesecond proposed navigational path.
 18. The method of claim 17, whereinanalyzing the at least one image includes identifying at least one tiretrack in the snow.
 19. The method of claim 17, wherein analyzing the atleast one image includes identifying a change of light across a surfaceof the snow.
 20. The method of claim 17, wherein analyzing the at leastone image includes identifying a plurality of trees along an edge of theroad.